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ABSTRACT 



This study investigated the effects of cognitive readiness in a Navy simulated 
environment, the simulation being the Multi-Mission Team Trainer. The research 
question that drove this study was: will simulations increase cognitive readiness? One of 
the tasks of Navy sailors is to deal with unpredictable events. Unpredictability in the 
military is considered to be one of the major characteristics, especially in battle. These 
officers must be prepared to react to these complex and unpredictable environments and 
simultaneously sustain competence in their performance. In this research such 
performances were driven by cognitive readiness. Increasing cognitive readiness amongst 
sailors can result in being able to recognize patterns in chaotic situations, modify problem 
solutions, and implement plans of action. 

This investigation was composed of a pilot study and a main study at the US 
Navy. The pilot study examined specific problem solving measures in simulation. The 
design involved measures before and after the Multi-Mission Team Trainer. All sailors 
(participants) took the cognitive readiness measures (domain independent measures) that 
were composed of a problem solving questionnaire (control strategies, self-efficacy, 
effort and perseverance, worry, and elaboration), teamwork questionnaire (adaptability, 
coordination, decision making, interpersonal, and leadership, and communication), and a 



creativity questionnaire 




(fluency, flexibility, originality, and elaboration). The measures from the pilot study 
were the same used for the main study. All scales have acceptable reliability. 

The main study included fifty four participants and three instructors who were 
given the same measures from the pilot study. The domain specific problem solving 
questions that were given to students included one retention, and one transfer question. 
Thus, the retention of participants was 24% and the transfer of participants was 9%. The 
significant cognitive readiness findings included higher levels of teamwork interpersonal 
and leadership skills, and creativity- elaboration (p=.05), following the simulation. Self- 
Efficacy and retention showed to be positively correlated. Creativity-flexibility and 
fluency resulted to be negatively correlated to transfer. This study could contribute to the 
understanding about increasing the cognitive readiness of US Navy sailors in order to be 
able to respond to unpredictable and complex events in a competent way. 




CHAPTER I: INTRODUCTION 
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Background of the Problem 

In the Navy sailors often train using simulations. Through simulations sailors are 
trained through realistic scenarios. The goal of this training is to increase their problem 
solving skills in order to transfer such skills to novel and real life situations. 

Unanticipated tactics, new technological capabilities, novel applications of existing 
technologies, and surprise are often characteristics of combat engagements (Fletcher 
2004). Since the most predictable characteristics of military operations is 
unpredictability, how do we prepare military personnel for the unexpected? (Fletcher, 
2004). 

The context for this research is in the Navy’s South Coast Naval School (SCNS- 
pseudonym). The goal of is to provide a continuum of professional education and training 
in support of Surface Navy Requirements that prepares officers to serve at sea. 

The South Coast Naval School offers a four level continuum of training from 
entry level officer training aboard ship up to senior officers assigned to major commands. 
The four levels of course are designed to train division officers, department heads, 
prospective executive officers, prospective commanding officers, and those selected for 
major commands. Each course is taught by officers of higher rank that have more 
experience and that require higher levels of knowledge, skills, and attitudes (Fox, 2007). 
The department head course is the second of the four levels of the South Coast Naval 




School courses and is intended for officers with mid-level responsibilities aboard ship. 
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This study focused on training sailors in air defense simulator and surface warfare - 
Multi- Mission Team Training. The Multi-Mission Team Trainer is a simulation used in 
the South Coast Naval School which provides tactical sensor, command, and control for 
simulations by ship and air combat. The simulator Multi-Mission Team Trainer also 
provides the necessary tools to train combat team supervisory personnel in the tasks that 
they must perform during the course of a mission. The Multi-Mission Team Trainer is 
designed to be a re-configurable multi-platform trainer. This study investigated the 
effects of cognitive readiness in a Navy simulated environment, the simulation being 
Multi-Mission Team Trainer. 

Purpose of the Study 

The purpose of the study was to increase the cognitive readiness of sailors who 
would be using the Multi-Mission Team Trainer. Cognitive Readiness is the mental 
preparation (including skills, knowledge, abilities, motivation, and personal disposition) 
an individual needs to establish and sustain competent performance in the complex and 
unpredictable environment of modem military operations (Fletcher, 2004). The Multi- 
Mission Team Trainer was adopted as the simulator due to the feasibility in the South 
Coast Naval School. The general problem in training is how to improve the results of 
training by providing the best training and assessment possible. 




Moreover, the definition that will be used for this research is that of O’Neil, 
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Perez, and Baker (in preparation). Cognitive readiness denotes the mental preparation for 
effective changes in response to altered situations in this fast- changing world. The term 
readiness denotes in the military a preparation or readiness to be effective in mission. 
Such term is used to distinguish combat readiness from combat effectiveness. O’Neil, 
Perez, and Baker (in preparation) view cognitive readiness through a knowledge, skills, 
and attributes (KSA) lens, i.e., knowledge is domain specific, skills are domain specific 
and domain independent, but attributes are relatively domain independent. 

Furthermore, simulations are important in the Navy setting because they outline 
the advantages of increasing safety, economy, environmental impact, and public 
relations. Furthermore, children and adults are required to interact with computers and 
simulators in all settings. Schools and organizations, and job training are finding that 
simulators are effective in reducing educational and training costs (Rifkin, 1994). There 
is also evidence that demonstrate that computer games and or simulations teach people 
more effectively (Cassidy, 2003; Jenkins, 2002). The frequency of computer games and 
simulation use, and the generated revenue show no signs of stopping or slowing. In 2004, 
$7.3 billion was spent on computer and video game software in the United States, 
breaking down eight titles every second of every day, according to the Entertainment 
Software Association (ESA) (2004). For these reasons it is imperative that we 
investigated the effectiveness of simulators, especially in the Navy. 




Significance of the Study 
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This study attempted to analyze the effects of cognitive readiness in a surface 
warfare simulation and provide feedback for the Navy’s Multi-Mission Team Trainer (air 
defense and surface warfare scenarios). The research on the effects of cognitive 
readiness in an air defense simulation and surface warfare simulation course was 
conducted in order to improve the problem solving skills in a Navy setting. In today’s 
world, it is imperative that the training that sailors receive is effective and contributes to 
force readiness. Effective training is critical. In the Navy, sailors train using scenarios 
within training simulations. In addition, the goal for this research was to increase the 
knowledge of the role of cognitive readiness and simulation. 
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CHAPTER II 
LITERATURE REVIEW 

The literature began by defining what simulation is, the relevancy to the research, 
and the areas that are unanswered. It is expected that simulations will increase cognitive 
readiness. For example, it is expected that the increase of simulation will increase content 
understanding, problem solving skills, and self-regulation. Furthermore, the literature 
review explores the relevancy of cognitive readiness, and problem solving. The literature 
review was conducted by searching in PsycINFO, ProQuest, and PsycARTICLES, from 
the years 1990 to current. In addition, the researcher limited the search to English only, 
peer-reviewed journal articles or book chapters from 1999-2008. In addition to using 
search terms to find articles the researcher also reviewed the reference of several 
dissertations. However, for the purpose of this research the researcher used mostly online 
journal articles. The terms that were included for simulation were: simulation games, 
computer games, simulation military, simulation Navy, and simulators. The articles that 
were chosen were empirical studies which dealt with the Navy setting. There were 
nineteen articles on simulation/simulators that were found. However, only eleven were 
used in the literature review due to the relevance to this study. The article, a research 
review Using Computer Games and Simulations for Instruction: A Research Review 
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(2006) by J.D. Fletcher and Sigmund Tobias provided a condensed review of the 
literature from 1992 to 2005. 



Simulation 

Definition of Simulation 

U.S. Navy units train in live, virtual, constructive, or mixed simulations of 
battlefield environments (Meliza, Stephen, and Goldberg, 2005). For the purpose of this 
study the definition that will be used, is the one stated by Gredler (1996). Simulations are 
a dynamic set of relationships among several variables. Simulations change over time and 
reflect authentic causal process. Many organizations are moving towards simulations due 
to the accessibility, flexibility, cost-effectiveness, and learning quality. The use of new 
technologies is justified by the premise that it can improve learner’s learning. Gredler 
(1996), suggests the simulations may be particularly appropriate for teaching about ill 
structured content domains. In these domains authentic, complex, and dynamic problems 
are encountered that require flexible access to knowledge from various sources. 

Moreover, much of the evidence emerging from evaluations and explorations of 
simulation suggests that teaching institutions while they are pursuing the specific goals of 
efficiency and flexibility, many of the resulting courses lack the informed design capable 



of providing enhanced learning. 
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Live simulation refers to the most realistic type of simulation. Live simulations 
entail real people using real equipment and performing in real venues. The training can 
be dangerous because the participants are almost exposed to the same risk as in real life. 
However, the scenarios can be scripted, resulting in lessening the danger. The rules of 
engagement also play a factor and prevent lethal use of force (Brooks et.al, 2004; 
Yardley, Thie, Schank, Galegher, & Riposa, 2003). 

Virtual simulation refers to real people operating simulated equipment in a 
simulated environment. Most of US military aviators receive their basic training in 
simulators. The last type of simulation is constructive. Constructive simulation refers to 
all elements being simulated, military personnel, equipment, and environment (Brooks et. 
al., 2004). For the purpose of this study, virtual simulation will be the focus. 

Why is the use of simulations important in training? The merits of simulations 
include facilitating learning by doing and triggering motivation. Also the engagement of 
learners in a simulated experience of the real world, makes learning practical (Mayer, 
Mautone, & Prothero, 2002). Educators and trainers began to take notice of the power 
and potential of simulators for education and training back in the 1970’s and 1980’s 
(2002). Simulators were hypothesized to be useful for instructional purposes and provide 
benefits such as; (1) complex and diverse approaches to learning processes and outcomes 
(2) interactivity (3) ability to address cognitive as well as affective learning issues (4) 
motivation for learning (O’Neil, Wainess, & Baker, 2005). 
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Instructors in the military, medicine, and business setting have applied 
simulators for training (Mayer et al. 2002). By using simulations, military personnel can 
practice flying and combat skills; medical students can practice diagnosis skills; and 
business leaders can practice fiscal, economic trading, and managerial decision-making 
skills (Dempsey, Haynes, Lucassen, & Casey, 2002). 

One of the major strengths of simulations is that they enable students to practice 
skills under realistic conditions and facilitate learning. Liberman and Linn (1991) argued 
that computer based simulations may be particularly useful in helping students to develop 
self-directed learning strategies, and in assisting students to apply knowledge in realistic 
settings O’Neil and Fisher (2004) conducted a literature review for games (mostly 
simulations) that concluded that simulations have the potential to facilitate learning in 
five ways: (1) enhance thinking skills (2) facilitate metacognition (3) improve knowledge 
and skills, (4) improve attitudes, and (5) promote motivation.. 

Dede (2005), also argued that learning in a well designed digital contexts can lead 
to the replication in the real world of behaviors in simulated environments. Schank 
(2005) agreed as well and stated that learning-by-doing is always more effective than 
learning-by-telling and that it is best accomplished through complex, high fidelity 
simulations that engage learners at the highest possible level. 

Effective problem solving in a simulation can potentially place a large cognitive 



load on working memory (Sweller, 2006). Thus there is a need to have a scaffolding 
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component, while there are a number of definitions, what they have in common is that 
scaffolding is an instrumental method that provides support during learning by reducing 
cognitive load. These processes provide learning goals, monitoring procedures, 
feedback, selection methods, hints, prompts, and various advance organizers (2006). 

Significance of Simulation 

Simulations are used in many environments as a mechanism to train personnel 
and increase effectiveness. Measurements should be embedded in the simulations, both to 
facilitate the design, administration, scoring, and reporting of traditional forms of 
instruction and to evaluate interventions that use technology to teach (Baker & O’Neil, 
2008). The long term benefit of combining technology and accountability is the intent to 
measure and report performance. Simulation also has other potential advantages, such as 
ensuring safety, overcoming training limitations, and protecting the environment. 

Maximizing skills, like problem solving through simulation training is the goal. In 
order to do so, knowledge acquisition and retention in extremely short periods becomes 
tractable if there is comprehensive and accurate information on the trainee’s background. 
The increase of metacognition skills could be potentially increased as a result of 
simulation. 

Another effect of simulation is the thinking skills of information processing, 
reasoning, inquiry, creation, and learning strategies (O’Neil, 1978). Simulation had a role 
to play in enhancing cognitive skills and processes that also apply in educational 
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situations (Pillay, Brownlee, & Wilss, 1999). Researchers have argued that simulations 
help improve students’ skills in practical reasoning (Wood & Stewart, 1987). 

Also, the development of knowledge and skills are improved through simulation 
training. Simulation training requires interaction with virtual-world situations and could 
potentially lead to the improvement of complex real-world motor skills such as driving 
on highways or piloting airplanes (Arthur et. al. 1995). The benefit of such training was 
based on the fact that learners were exposed to various simulated situations, they 
developed knowledge of the different domains (Maguill, 1993). 

Assessment of Simulation 

There are multiple ways of assessing learning through simulators. O’Neil, 
Wainess, and Baker (2005) generated suggestions in a game context which can also be 
applied to simulation. For example, one could assess the training effects of a game by 
examining trainee’s ability to solve criterion problems, their application of declarative 
and procedural knowledge, willingness to raise or lower game challenge, their self- 
reports, and records of their play. Evaluation questions to be answered about the 
cognitive and affective effects of games or simulators should concern the four levels of 
Kirkpatrick (1994). 

Kirkpatrick’s four levels of evaluation are known in the training industry. A four 
model consisting of (1) reaction, (2) learning, (3) behavior, and (4) results. According to 
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this model evaluation should always begin with level one and progress as time and 
money allows it, and move sequentially (Hoffman, 2004). Kirkpatrick described a four- 
level evaluation system that handles most of the questions anyone may want to ask about 
training systems. For the relevance of this study only two levels of Kirkpatrick might be 
used to evaluate the effectiveness of a simulator. Level 1, reaction and level 2, learning 
can be used as an evaluation tool. 

Level 1 is reaction, the trainee’s reaction to the program and their level of 
satisfaction. Level 2 is learning, the extent in which participants change attitude, improve 
knowledge, and increase skills (Mehrotra, 2001). Level 3 is behavior, occurs after the 
training, and is the extent in which change in behavior has occurred because the 
participant has attended the training program. Level 4, results, refers the organizations 
benefits and the final results due to the participant attendance (Clark & Estes, 2002). 

Level 1 Reactions 

Reaction evaluation is usually conducted with a few questions that ask 
participants whether they like and value the program. These questions can be asked 
before a program starts, during a program, and at the end of a program. Level 1 indicates 
people who are motivated to persist and to invest effort in the performance program. 
Positive reactions do not indicate whether they gained anything useful or will use 
information after the program is finished. Reaction information also does not tell you 



whether a successful program will support the organizational goals. The benefit of Level 
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1 information is that it tells you about the motivational impact on the participants 
(Mehrotra, 2001). 

An example, is the study conducted by Baker, Prince, Shrestha, Oser, & Salas 
(1993), which surveyed 112 military aviators and their reactions to the Crew Resource 
Management simulation. Their reactions included that they felt good about learning, and 
most agreed on the importance of the Crew Resource simulator. 

Level 2 Learning 

Level 2 evaluation targets the impact of all programs while they are being 
implemented. If a knowledge gap is being closed with a training program, this level 
examines the learning that takes place during the training course. If a performance 
improvement gets off track, Level 2 serves as an early warning and an opportunity to 
make corrections. The assessment should be based on the task analysis used to design the 
training. 

The evaluation conducted by Helmreich, Merritt, & Wilhelm (1999), which 
surveyed pilots in several organizations after completion of Crew Resource Management 
simulator stated that in level 2, learning, no specifications of skills were taught, decay in 



attitudes were not immediately apparent but some decay over time. 




Level 3 Behavior 
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Level 3 is applying what the learner has acquired to real practice or on the job. 

The evaluation designed ensures that the program or training has a positive influence on 
job performance, for example a training transferring to a job. 

Level 4 Result of the Program 

Level 4 evaluation-results, is the highest level of evaluation in Kirkpatrick’s 
(1994) evaluation framework, and most complicated. Salas (2001) pointed out the 
complexity of this level. Evaluators are looking for evidence that the programs have 
influence the organization by comparing pre and post test or experimental and control 
groups, to recognize the possibility that some variable s could have contributed to the 
results. For example, applying Kirkpatrick’s framework for evaluating an aircrew training 
program (Salas, 2001), the researcher reviewed 58 published accounts of training 
programs to determine its cost-effectiveness, and found the results uncertain. However, as 
pointed out by the researcher (Salas, 2001), evidence shows the effectiveness of the 
program had been found, such as reduction in accident rates. 

Summary of Simulation 

Simulations are important in the Navy in order to facilitate learning and training. 
The literature suggests (O’Neil & Fisher, 2004) that simulations enable students to 
practice skills under realistic scenarios. As a result, simulations create a very important 
tool in the training of Navy personnel. The positive outcome of having simulators is that 
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it can increase the metacognition of sailors, develop level knowledge of different 
domains and be exposed to different environments. Moreover, simulations can assist the 
students in the assessment of performance. For example, simulations can track the 
student’s ability to solve criterion problems, application of knowledge, self reports and 
records of their play. 

Simulations can enable sailors to practice skills under realistic conditions in 
which can facilitate their learning and increase their cognitive readiness. A possible 
example, is that sailors can practice flying and combat skills and as a result be able to 
competently react to unpredictable environments. In conclusion, the literature finds 
(Sweller, 2006), that simulations provide learning goals, monitor procedures, provide 
feedback, prompts, assessments, and various advance organizers. Therefore, resulting in a 
crucial tool in the Navy to train and assess sailors. Furthermore, this study hypothesizes 
that simulators will increase cognitive readiness. The next section will define cognitive 
readiness and the relevancy to simulators. 

Cognitive Readiness 

This section reviewed the definition of cognitive readiness, the competencies of 
cognitive readiness that were used for this research (eg. problem solving, decision 



making, teamwork, and creativity), and the significance and assessment of cognitive 
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readiness. This study measured the effects of cognitive readiness by incorporating 
measures in a simulation environment. 

Definition of Cognitive Readiness 

Cognitive readiness is the mental preparation (including skills, knowledge, 
abilities, motivations, and personal dispositions) an individual needs to establish and 
sustain competent performance in the complex and unpredictable environment of modem 
military operations (Fletcher, 2004). Moreover, cognitive readiness can be conceived as a 
set of three basic abilities: 1. recognize patterns in chaotic situations, 2. modify problem 
solutions associated with these patterns as required by the current situation, and 3. 
implement plans of action based on these solutions (Fletcher, 2001). 

Table 1 (below) has been adopted from Morrison and Fletcher (2001) and 
identifies ten competencies that form a basis for cognitive readiness. Competencies are 
defined as “sets of behaviors that are instrumental in the delivery of desired results or 
outcomes”(Bartram, Robertson, & Callinan, 2002, p.7). The competencies include: 1) 
leading and deciding, 2) supporting and cooperating, 3) interacting and presenting, 4) 
analyzing and interpreting, 5) creating and conceptualizing, 6) organizing and executing, 
7) adapting and coping, and 8) enterprising and performing (Bartram, 2005). 



Table 1: Fletcher’s Cognitive Readiness Competencies 





Situation Awareness 



Memory 



Transfer 



Meta-cognition 



Automaticity 
Problem Solving 
Decision-Making 
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It is defined as the ability to 
perceive and comprehend oneself in 
relationship relevant to present 
environment (Endsley, 1988). It is 
measured by simulated operations. 

Practice and feedback in complex 
simulated environments have 
improved situation awareness. 



Able to recall and/or recognize in the 
current operational situation patterns 
that will lead to likely solutions. The 
two theoretical mechanisms are 
encoding specificity and transfer of 
appropriate processing. 



Being able to apply what is learned 
in one context to a different 
performance context. 



Execution function of thought to 
monitor, assess, and regulate one’s 
own process. 

Process that requires only limited 
conscious attention. 

Ability to analyze the current 
situation. 

Emphasis is on recognizing learned 
patterns and reviewing courses of 
action. 




Mental Flexibility and creativity 



Ability to generate, adapt, modify, to 
variable situations. 
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Leadership 



Emotion 



Combination of technical, 
conceptual, ethical, and interpersonal 
competencies that encourage support 
to others. 

Able to perform complex tasks under 
the stress and confusion of modern 
military operations. 




Moreover, O’Neil (in preparation) modified Fletcher’s model (see figure 1). 
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The O’Neil Cognitive Readiness Model provides a comprehensive framework for 
understanding, training, and evaluating cognitive readiness (O’Neil, Perez, and Lang, in 
preparation). In this model, three key constructs are shown; knowledge, attributes, and 
strategies. The knowledge includes domain- specific knowledge for developing cognitive 
readiness in the specific domain. There are eight competencies or skills: situation 
awareness, adaptive expertise, creativity, resiliency, problem-solving, adaptability, 
decision-making, and teamwork. This study will focus on four competencies that 
specifically apply to the South Coast Naval School setting: problem solving, decision 
making, creativity, and teamwork. 




Figure 1: CRESST Cognitive Readiness Model (in preparation) 
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Table 2 defines the competencies of CRESST’s cognitive readiness model (in 
preparation). However, four of the competencies (problem solving, decision making, 
teamwork, and creativity) were the focus for this research (O’Neil, in preparation). The 
bolding in Table 2 indicates competencies investigated in this research. 
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Table 2: CRESST’s Cognitive Readiness Competencies 



Adaptive Expertise 


Entails a deep comprehension of 
conceptual structure of the problem 
domain. Knowledge must be 
organized and structures must be 
flexible. Adaptive experts 
understand when and why particular 
procedures are appropriate or not 
(Zaccaro & Banks, 2004). 


Creativity 


Includes the sensitivity to problems, 
fluency, novel ideas, flexibility, 
synthesizing, analyzing, complexity, 
originality, elaboration, and 
redefinition (O’Neil, Abedi, & 
Spielberger, 1994). 


Resiliency 


A class of phenomena characterized 
by patterns of positive adaptation in 
the context of significant adversity 
or risk (Masten & Reed, 2002). 


Situation Awareness 


Ability to perceive and comprehend 
oneself in relationship relevant to 
present environment (Endsley, 
1988). 


Problem- Solving 


Problem solving is content 
understanding, problem solving 
strategies, and self-regulation 
(O’Neil, 1999). 


Adaptability 


An effective change in response to 
an altered situation (Mueller- 
Hanson, 2005). 


Decision-Making 


Use of information about their 
current situation to help evaluate the 
utility of potential courses of action 
(O’Neil, Chung, and Herl, 1999). 
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Teamwork 


A trait of the individual that 




predisposes the individual to act as a 




team member. There a six processes: 




(a) adaptability, (b) coordination, (c) 




decision-making, (d) interpersonal, 




(e) leadership, and (f) 




communication (O’Neil & Baker, 




2003). 



Teamwork/ Decision Making 

The measurement approach is to view teamwork skills as a trait of the individual 
that predisposes the individual to act as a team member (O’Neil, Wang, Mulkey, and 
Baker, 2003). A trait is a characteristic in a person that is relatively enduring (intelligence 
or personality). O’Neil, Wang, Mulkey, and Baker (2003), state that there are various 
definitions of teams. However, they state that the following are useful ones: (a) Teams 
are composed of two or more people who share a common goal, (b) Teams are composed 
of members of a working group identified as a team, (c) A team is a set of two or more 
people who interact dynamically, interdependently, and adaptively in working toward a 
common and valued goal who have been assigned specific roles and have limited life 
span of membership (Baker & Salas, 1992). 

The taxonomy (O’Neil, Chung, & Brown, 1997) is made up of six teamwork 
processes. These skills include (a) adaptability- recognizing problems and responding 
appropriately; (b) coordination- organizing team activities to complete a task on time; (c) 
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decision making- using available information to make decisions: (d) interpersonal- 
interacting cooperatively with other team members; (e) leadership-providing direction for 
the team: and (f) communication- clear and accurate exchange of information. 

Teamwork skills included decision making. Decision-making capability affects 
the ability to capitalize on available information. Effective teams use information about 
their current situation to help them evaluate the utility of potential courses of action. 
Effective individuals and teams employ decision making that takes into account all 
available information (O’Neil, Chung, and Herl, 1999). It is expected that decision 
making processes emerge when the nature of a relationship is unknown. Members need to 
assess the quality of the relationship based on their knowledge of the domain. It is 
expected that content knowledge play a significant role in how decision-making process 
play out. 

Creativity 

According to O’Neil, Abedi, & Spielberger (1994), there are many definitions of 
creativity but the one that was adopted is by Torrance (1989). Torrance has been involved 
in teaching and the measurement of creativity. According to Torrance (1989) creativity 
includes sensitivity to problems, fluency, novel ideas, flexibility, synthesizing, analyzing, 
complexity, originality, elaboration, and redefinition. Some of the factors underlying 
Torrance’s definition of creativity and the Test of Creative Thinking are adapted from 



Guilford’s Structure of Intellect Model. 




Based on the constructs of the Torrance Test of Creative Thinking (TTCT), a 
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multiple-choice paper- and- pencil test was developed. The purpose of developing this 
new instrument was to shorten the amount of time required for the administration and 
scoring of creativity tests. Abedi and Schumacher (2002) constructed a 60-item multiple 
choice test to measure creativity. The reliability and validity of this test were examined in 
the study conducted in Spain on a group of 2,270 students (Villa, Auzmendi, & Abedi, 
1996). This study used data from the Spain Study (Villa, Auzmendi, & Abedi, 1996) and 
employs a latent-variable modeling approach for assessing reliability and concurrent 
validity. 



Significance of Cognitive Readiness 

The importance of cognitive readiness derives from the characteristics of 
unpredictability that the military personnel face in battle. According to Fletcher (2004), 
unpredictability is the most predictable characteristic of military operations. The 
cognitive readiness vision is to optimize the human contribution to joint war fighting, and 
achieve the revolutionary war-winning capability articulated in the Department of 
Defense Joint Vision 2010. According to the Department of Defense achieving cognitive 
readiness will ensure that the war fighter is mentally prepared for accomplishing the 



mission, the war fighter is performing at his/her optimal performance level, the tools and 
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techniques for preparing the war fighter are the most effective and affordable (Etter, 
Foster, and Steele, 2000). 

Fundamentally, cognitive readiness focuses the science and technology efforts on 
addressing the critical need for increased capability and adaptability from the human 
component on weapon systems in a progressively more complex, dynamic, and resource- 
limited environment (Etter, Foster, and Steele, 2000). The joint war fighter is challenged 
by the potential of simultaneous, multiple, geographically separate, high-or-low intensity 
conflicts, as well as peacekeeping, counterterrorism, and disaster support missions. 

Assessment of Cognitive Readiness 

Cognitive readiness skills are trainable and assessable. O’Neil, Perez, and Fang 
(in preparation), provide an example of how situational awareness can be improved by 
conducting practice and feedback in complex, simulated environments. Metacognition 
can be improved by exercises designed to increase awareness of self-regulatory processes 
(Hacker, 2001). Problem solving abilities in many tasks can be improved through practice 
with feedback and learning (Fletcher, 2004). 

Moreover, CRESST Cognitive readiness model (in preparation) can be used as a 
framework for model-based assessment. Each of the three key constructs of cognitive 
readiness- knowledge, attributes, and strategies- can be assessed individually. The 
assessment of the three constructs (e.g., the domain- specific knowledge, the 



competencies, and the specific skills) may denote cognitive readiness and can predict 
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performance (O’Neil, Perez, and Lang, in preparation). An example, that Bartram 
(2005) provides is that of the competencies being a framework for making predictions 
from measures of competency potential (ability, personality, and motivation) to ratings 
of actual work performance. Each of the eight predictors is shown to predict different 
area of job performance consistently across jobs, measurement instruments, and cultural 
contexts. 

Some of the studies that O’Neil, Perez, and Lang ( in preparation) stated which 
have evaluated cognitive readiness are that of Sommer, Whitman, Borkowski, Gondoli, 
Bruke, Maxwell, & Weed (2000). Another study conducted by Wood, Lugg, Hysong, 
and Harm, (1999) which dealt with psychological adaptation to extreme environments 
used questionnaires and a sliding scale ranging from 0 to 100 and open-ended questions. 

Fletcher (2004) pointed out that many aspects of cognitive readiness can be 
measured and assessed but need systematic development. Fletcher (2004) points out 
that the reliability, validity, and the precision for the evaluation of cognitive readiness 
need to be developed. 

Summary of Cognitive Readiness 

In order to prepare units and individuals to deal with unexpected scenarios, 
cognitive readiness must be in place. Through cognitive readiness learners will be 
able to recognize patterns, modify problem solutions, and implement plans of action 



based on these solutions. Cognitive readiness provides a framework for understanding, 
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training, and evaluating. The key constructs that determine cognitive readiness are 
knowledge, attributes, and strategies. The main advantages of cognitive readiness and its 
competencies is that it also provides integrating measures in the domain, such as ability, 
personality, and motivation scales (Bartram, 2005). 

Problem Solving 

Problem solving is an important skill that is needed in training using simulations. 
Problem solving is one of the family of cognitive demands that can be required in many 
subject areas (Mayer, 1996). Problem solving skills fill a gap by allowing students to use 
what they have learned to successfully solve new problems or leam new skills. Mayer 
and Wittrock, (1996), state that problem solving is the “cognitive processing directed at 
achieving a goal when no solution method is obvious to the problem solver” (p.3). 

Definition of Problem Solving 

A problem exists when the solver has a goal but does not know how to reach it, 
so problem solving is mental activity aimed at finding a solution to a problem (Baker & 
Mayer, 1999). Problem solving has four characteristics: (1) cognitive- because it occurs 
in the problem solver and can only be inferred from behavior; (2) process-based because 
it manipulates or transforms knowledge; (3) directed- because it is intended to help 
achieve a goal that is not directly achievable, and (4) personal- because it depends on the 



existing knowledge and skills of the problem solver (Baker & Mayer, 1999). 




The CRESST model of problem solving (1999) is adapted from the problem- 
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solving models of Glaser, Raghavan, and Baxter (1992) and Sugrue (1995). There are 
three components of problem-solving, adopted from O’Neil (1999, 2007), which is the 
theoretical framework for this study as may be seen in figure 2. This model is analyzed 
into three components, content understanding, problem solving strategies, and self- 
regulation (O’Neil, 2007 revised). 
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Figure 2: O’Neil’s Problem Solving Model (in preparation) 




Test Anxiety 
(Worry) 
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According to this model, a successful problem solver sailor needs to possess: (a) 
content knowledge- understand something, (b) problem solving strategies- have skills to 
solve problems, and (c) metacognition- plan and monitor problem solving progress, and 
(d) have motivation to solve problems (O’Neil, 1999). 

The definitions of each construct for the O’Neil problem solving (in 
preparation) model are: Content understanding. In this problem solving model, the 
content understanding is the understanding of domain knowledge. It is defined as a 
person’s ability to understand significant facts, procedures, concepts, and principles 
related to the domain knowledge of content (O’Neil, 1999). 

Problem solving strategies. Problem solving strategies are categorized into domain- 
specific problem solving strategies and domain-independent problem solving strategies. 

Domain- specific problem strategies are task-dependent strategies. Domain-specific 
problem solving strategies are strategies in a particular field of study or subject, such as 
the application of the tactical strategies (surface warfare) in the Multi-Mission Team 
Trainer question (Chen, 2005). In this study, domain- specific problem solving strategies 
will be the strategies sailors acquire to successfully complete the Multi-Mission Team 
Trainer. The following are the retention and transfer question. 

Retention Question: 

• Write an explanation of how you solve the tactical problem scenario. 
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Transfer question: 

• Write some ways of improving the scenario used in the Multi-Mission Team Trainer. 

Domain-independent problem strategies are general strategies. Domain- 
independent problem solving strategies are strategies that are not linked with a specific 
domain. Baker and O’Neil (2002) state, “Domain-independent analyses focus on the 
subject matter as the source of all needed information... domain independent analyses are 
those that attempt to capture the general strategies that are in us across subject matters” 

(p. 619). 

Self -regulation. Self-regulation refers to the extent that learners are metacognitively, 
motivationally, and behaviorally active in the learning process to control their learning 
and reach their goals (Zimmerman, 1989). In this problem solving model, self-regulation 
is composed of metacognition and motivation. Metacognition includes planning/self- 
monitoring and motivation contains effort, self-efficacy, and test anxiety (worry). 

In order to be effective in a simulation the process of self- regulation must take place 
in order to increase skill acquisition. Learners might not automatically self-assess, 
therefore one way is to have them periodically self-assess their progress. By making 
performance improvements salient, such monitoring should raise self-efficacy, sustain 
self-regulation, and promote skills (Pintrich & Schunk, 2002). Self-assessment enhances 
goal setting effects on performance when goals are informative of one’s capabilities. In a 



study conducted by Schunk and Ertmer (1999), self-assessed learners were able to 
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produce comparable results when coupled with a process or product goal. Constant 
self-assessment can raise achievement outcomes. 

Self-regulated learners are proactive to incorporate various self-regulated 
processes (e.g. goal setting, self observation, self-evaluation) with task strategies and 
self-motivational beliefs. Students who have been trained in self-regulation processes 
during learning, such as goal setting, self-monitoring, and self-reflection processes, 
display high levels of motivation and achievement (Shunk, 1996; Wook, Bandura, & 
Bailey, 1990). 

Planning and self-monitoring are important characteristics of self-regulated 
learners. Self-regulated learners usually plan how to effectively check their progress 
(Zimmerman, 1998). When it is necessary, they will change their learning strategies or 
modify their goal (Zimmerman, 1998). 

Motivation is defined as “the process whereby goal-directed activity is instigated and 
sustained” (Pintrich & Schunk, 2002, p. 5). Motivation is something that instigates people 
to learn, directs people to move forward, and sustains people to keep going on (Ormrod, 
2006). 

Effort is one significant index of motivation. Motivated sailors usually expend more 
mental effort to leam difficult material. Effort and self-efficacy are strongly correlated. 
Sailors who have high self-efficacy beliefs would like to spend more effort when a task is 



perceived difficult (Pintrich & Schunk, 2002). 
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Self-efficacy is identified by Bandura (1986, p.391) as “people’s judgments of their 
capabilities to organize and execute course of action to attain designated types of 
performance.” Self-efficacy beliefs contribute to motivation as well as student learning 
achievement. Students with high self-efficacy beliefs have a higher level of motivation 
and better achievement (Zimmerman, 1990). 

Test anxiety refers to phenomenological, physiological, and behavioural responses to 
thoughts about negative events or consequences such as failure on an exam or an 
evaluative situation (Speilberger,1980). The phenomenological responses are categorized 
into two components: worry (cognitive) component and emotional (affective) component 
(Speilberger, 1980). Worry is the cognitions associated with test anxiety. When sailors 
have high test-anxiety, they are more likely to worry and will not be able to answer a 
question in a test or finish the test. 

Significance of Problem Solving 

An important conclusion from the researched literature (O’Neil, 1999) is that 
problem solving skills should be learned within the context of realistic problem-solving 
situations. This approach suggested the modeling of how and when to use strategies in 
realistic tasks. As technology becomes more integrated with problem solving it becomes 
increasingly more important to find ways of evaluating what students are learning 



through simulations. 
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Assessment of Problem Solving 

According to Mayer (2002), problem solving is one of the most significant 
competencies whether in job settings or in schools, and, as a result, teaching and 
assessing problem solving has become one of the most significant educational and 
training objectives. Mayer (2008) argues that there are two forms of assessments: (a) tests 
of conceptual knowledge, including open-ended verbal questions involving 
troubleshooting, redesigning, and explaining; and (b) tests of strategic knowledge, 
including behavioral assessment of learners’ strategies in solving transfer problems. 
Research in cognitive science suggests that problem-solving performance depends on the 
coordination of several different kinds of domain specific knowledge, including facts, 
concepts procedures, and strategies (Anderson, et. al., 2001). 

O’Neil (1999) has pointed out that problem solving is a critical competency 
requirement of college students and employees. As a result, consistent and high quality 
assessments for problem solving skills are of importance. Reports or findings on problem 
solving assessment are thought to help trainers and organizations to assist in allocating 
resources to meet particular gaps (Baker & Mayer, 1999). 

The CRESST model of learning. Baker (1995) posits five families of cognitive 
learning: content understanding, collaboration, communication, problem solving, and 
metacognition. The five families describe the range of cognitive learning in which 
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learners engage; they are seen as working together to influence overall learning. 

Content understanding is the first type of learning of subject matter content. 

Assessment of content understanding should evaluate not only basic factual 
knowledge, but also a deeper level of understanding of the subject area (Herman, 

1992, Linn, Baker & Dunbar, 1991). 

These assessment models includes the following activities: simulating prior 
content area knowledge, reading primary source documents containing new 
information, and writing an explanation of important issues that integrates new 
concepts with prior knowledge. Understanding is assessed by examining overall 
content quality, prior knowledge, principles, and use of resources (Klein, O’Neil, 

Dennis, Baker, 1997). 

The value of assessing problem solving- processes is that data on such 
processes can provide evidence of what a learner is doing while carrying out a task. 
Process evidence can be used, to help evaluate the extent to which a task evokes 
expected problem-solving behaviors, help explain performance differences between 
subgroups, or aid in task validation. Measures of problem solving processes, when 
used in accordance with measure of knowledge and problem solving performance, 
can provide a more comprehensive picture of the learner. 

According to Baker & O’Neil (2002), in order to improve quality of 



measurement in problem solving, four things must be considered: (1) intentions and 
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skills of assessment designers, (2) the range of performance that counts as problem 
solving; (3) ways in which validity evidence can be sought (4) the degree in which 
measurement produces results that generalize across tasks and contexts. 

Problem solving can be a focus of computer interventions that are dedicated to 
a particular content area and for which appropriate solutions are known in advance 
(e.g. to land an F-14 on a carrier deck through simulation), where the goal is to 
determine whether the trainee has enough specific knowledge and strategies to 
accomplish one or more tasks needed to meet a known standard. The approach to 
solving problem is to determine the characteristics of challenging conditions (e.g. 
high seas for the aircraft carrier deck landing) and figure out which of a series of 
procedures to apply (Baker & O’Neil, 2002). A problem solving task may be very 
difficult, the learner may need strategies to apply in order to recover from error (e.g. 
danger to an aircraft or ship). 

Another approach to the assessment of content understanding is to evaluate 
learner’s underlying knowledge structures. Research on memory suggests that 
knowledge is complex semantic networks (Jonassen, Beissner, & Yacci, 1993: 

Rumelhart & Ortony, 1977). By eliciting learner’s specific information from students we 



can attempt to assess these cognitive structures. 
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Measurement of Content Understanding 

In the cognitive theory of learning, it is stressed that learned knowledge should be 
organized into long-term memory for later access (Mayer, 2008). The expertise literature 
suggests that experts’ understanding of domain knowledge is awareness not only of the 
concepts but also of the connections among the concepts (Schau & Mattern, 1997). To 
assess content understanding CRESST has used both essay-based explanation tasks (e.g. 
Baker et. al., 1995) and knowledge mapping tasks (1995). A concept map is a graphical 
representation of information consisting of nodes and links or labeled links. Nodes 
correspond to concepts within a particular subject area or domain: links indicate 
relationships between pairs of concepts (or nodes), and labels on each link explain how 
two concepts are related. Learners create concept maps by identifying important concepts 
and generating and appropriately labeling the links between those concepts. This 
approach assumes that a deep understanding in a subject domain allows an individual to 
conceive a rich set of interrelationships among important concepts within that domain. 

Simulators have the ability to capture process differences (Chung & Baker, 1997; 
Schacter, Herl, Chung, & O’Neil, 1999; Stevens, Ikeda, Casillas, Palacio-Cayetano, & 
Clyman, 1999) and provide a pattern of emerging competence based on time, strategy, 
and response of the learner. Second, simulators have the ability to make difficult-to- 
perceive processes visible. Third, simulators can provide online scoring and feedback. 
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Measurement of Problem Solving Strategies 

Problem solving has to have an intuitive component (continual monitoring and 
understanding of context and changes) and a resulting level or degree of automaticity. 

In dealing with simulations, problem solving can be in a third form, dealing with 
simulations and problems for which there is not a known solution, but present a fast 
changing scenario, for instance, with change of probability of existing faults occurring as 
“surprises” during the examination sequence. In this case the students, incorporates 
useful strategies in order for procedures or actions to optimize the outcome. 

In simulations, learners can also demonstrate their expertise through certain tasks. 
In this environment the learner can submit the articulation of problem and receive 
feedback. The learner can apply their strategies and procedures to solve the problem 
(Baker & O’Neil, 2002). 

Summary of Problem Solving 

Problem solving strategies help students analyze a situation, help solve new 
problems and/ or learn new skills. The literature states (Mayer, 1990), that problem 
solving is the process of transforming a given situation into a desired situation when no 
obvious method of solution is available to the problem solver. The model that has been 
adopted for this proposal is the O’Neil’s Problem solving model (revised 2007). The 
three components that will be analyzed are content understanding, problem-solving 



strategies, and self-regulation. The literature also concludes (O’Neil, 1999) that problem 




solving skills should be learned within the context of realistic situations. Therefore, 
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simulators are a vehicle to display such situations. 

Problem solving skills are an imperative component in understanding the 
capabilities of sailors in the Navy. Therefore, it is important for sailors not only to 
acquire problem solving skills but also to be able to measure the process. The assessment 
can be both, evaluation of the search and evaluation of knowledge integration (Baker, 
et.al., 1995). According to Chung & Baker (1997), simulators can capture problem 
solving strategy differences and provide patterns of emerging competence based on time, 
strategy, and response of the learner. The goal is for sailors to increase their problem 
solving skills and transfer their capability to live scenarios. 
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CHAPTER III 
Methods 

Research Question 

Research Question: Will a simulation increase cognitive readiness? 

Hypothesis 1: There will be an increase of scores between the pre and post cognitive 
readiness questionnaire. 

Research Design 

In February 2008, the researcher conducted her study at the South Coast Naval 
School. In addition to her study, there was another study conducted simultaneously that 
focused on the formative evaluation of the Multi- Mission Team Trainer. However, this 
study focused on a specific task (surface warfare scenario) to test the cognitive readiness 
of sailors before and after the Multi- Mission Team Trainer. 

The study was composed of two phases: a pilot study and a main study. The pilot 
and main study took place after it was approved by the USC Review of Human Subject 
(see appendix A). The pilot study was conducted in January 2008 and the main study was 
conducted in February 2008. The design involved measures before and after the Multi- 
Mission Team Trainer. In addition, the feasibility of the study procedures was 
investigated in order to make sure that lessons learned would be implemented for the 
main study. All students took a set of questionnaires that were given to assess the 
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environment, the feasibility and understandability of the procedures for participants 
and instructors. The questionnaire included three constructs of cognitive readiness. The 
four constructs were problem solving, teamwork, decision making, and creativity. There 
were a total of 1 17 items. The number of participants in the main study was fifty-four. 

However, only fifty-two students out of the fifty-four took the open-ended 
questions as they stated they had a high stakes test to take (main study). In addition, three 
expert instructors were also given the open- ended questions that tested retention and 
transfer knowledge. The questions specifically pertained to the cumulative surface case 
scenario. 



Pilot Study 

The pilot study focused on one specific simulation, the Multi-Mission Team 
Trainer (MMTT). The scenarios were provided from the air defense module. There were 
three students participants who filled out the questionnaire and two instructors. The 
purpose of the pilot study was to understand if the environment was feasible and 
understandable for the participants and instructors, and asses the time that would take to 
fill out the questionnaire. Finally, the pilot study was conducted to find out participants’ 
cognition towards the Multi-Mission Team Trainer and the entire simulation process. 

The cognitive process mapper was going to be an alternative in case that the Multi 



Mission Team Trainer would not be feasible. The cognitive process mapper was 




developed by CRESST (National Center for Research on Evaluation, Standards, and 
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Student Testing) is considered to be an advancement of concept maps. It is a simulation 
for assessing content knowledge. The cognitive process mapper is composed of three 
sections, 1. data repository, 2. decision inventory, and 3. action item. For each item a 
value can be assigned indicating how important that particular datum or relationship is. 
This provides valuable assessment data on what the learner “thinks” about the 
information, and can elicit important errors in thinking, such as decision making biases 
(Wainess, 2007). However, the cognitive process mapper was not used. 

Students were asked to create concept maps using software on a standalone 
personal computer. Figure 3 is a table of concepts produced and given to students. The 
items in the concept list are reduced per the guidance of the South Coast Naval School. 
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Figure 3: Table of Concepts 

Build the map in five steps. 

1 . Place major Events in the appropriate distance region (far. medium, close). 

2. Place Information Sources and connect them to the appropriate Events or other 
Information Sources. 

3. Place Purposes and link them to the appropriate Events. 

4. Place Tools used to implement events and link them to the appropriate Events. 

5. Add Actors and link them to the appropriate Events. Information Sources, or 
other Actors. 





CONCEPT 






RELATION 


Event 




Information Sources 




Purpose 








Engagement 




Air Route 




Employ Chaff 






conducts 


ID 




Altitude 




Reduce RCS 






determines 


Intent 




Change 




Unmask Batteries 






has 


Maneuver 




Country of Origin 






orders 




ECM 




Tools 






part of 




IFF 




CIWS 






recommends 




Illuminate (response to) 




Decoys 






supervises 




I&W 




Guns 






to 




O/S Caps and Lims 




Surface to Air Missile 






uses 




Profile 








Query 




Actors 








Speed 




AIC 








Threat Capabilities 




Air 








VID 




DC A 








Warn (response to warning) 




OOD 








Wind 




TAO 








Wings Call 







However, the Multi Mission Team Trainer ended being the most feasible 
simulation for this study. 

Multi-Mission Team Trainer (MMTT) 

The Multi-Mission Team Trainer (MMTT) provides tactical sensor and command 
and control (C2) simulation for use by ship and ship/air combat teams and strike group 
staff supervisory-level personnel. The Multi-Mission Team Trainer system supports a 
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wide range of training operations, it can be run as a stand-alone multi-ship/multi- 
aircraft trainer, or can be interfaced to existing training devices to upgrade or enhance 
current training capabilities. 

The Multi-Mission Team Trainer system provides the necessary tools to train 
combat team supervisory personnel in the tasks that they must perform during the course 
of a mission; Develop the decision-making and communications skills of ship and 
ship/air combat teams and strike group supervisory-level personnel. The Multi-Mission 
Team Trainer is designed to be an easily re-configurable multi-platform trainer. It uses 
software modules to simulate generic shipboard and aircraft Combat Systems and 
console. Wherever practical, operating procedures for trainee stations have been modeled 
after procedures and formats implemented in the actual equipment that the station is 
designed to represent. The following are case scenarios given to students via the Multi 
Mission Team Trainer (see table 3) . For the purpose of the pilot study the air defense 
scenarios were adopted due to their feasibility. The air defense case scenarios were 
acquired from a colleague who had several experiences observing the Multi Mission 
Team Trainer and who had been working with the South Coast Naval School in various 



projects. 




Table 3: Multi-Mission Team Trainer- Case Scenario- Air Defense 
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Scenario 1: 


Commercial flight then a bad guy 
ID the commercial track 
ID the bad guy 
Shoot the bad guy down 


Scenario 2: 


Commercial and two bad guys in sequence 

ID the commercial track 

ID the first bad guy 

ID the second bad guy 

Reports 

Shoot the bad guys down 


Scenario 3: 


Same as scenario 1, but have an FI 8 
ID the commercial track 
ID the bad guy 

Send the F-18 for VID, escort, and kill 
Reports 

Shoot the bad guy down 


Scenario 4: 


Same as scenario 2, but have an F-18 
ID the commercial track 

ID the first bad guy ( a MIG- 21, not a threat to the MEU) 

Send the F-18 for VID and escort- ideally no kill because 

the second guy is an MEU threat 

ID the second bad guy (SU-27, a threat to the MEU) 

Reassign the F-18 for VID, escort, and kill 

Reports 

Shoot the bad guy (s) down- ideally, the SU-27 


Scenario 5: 


Littoral and close to shore, so time pressure, peacetime, 
ideally don’t shoot, two groups of three ships, one with an 
MEU 

ID commercial tracks, including Cessna 
Report 
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Table 4 explains the different acronyms and abbreviations in the air defense 
case scenarios. 

Table 4: Acronyms and Abbreviations 



Terms 


Definitions 


F-18 


Airplane- Fighter (United States) 


VID 


Visual Identification 


MEU 


Mission Essential Unit 


SU-27 


Russian Design Jet Fighter 


ID 


Identification 


MIG-21 


Russian Design Jet Fighter 



Participants 

The participants consisted of three navy sailor students and two instructors. The 
three student participants completed the 117-item questionnaire and provided feedback 
on the questions. They also responded to the open-ended retention and transfer question 
on air defense case scenarios. The purpose of interviewing the two instructors was to 



obtain an expert opinion. 





47 



Measurement 

Part of our methodology was to assess if sailors will demonstrate better cognitive 
readiness after a simulation. Therefore, a number of instruments were adopted for various 
measurement purposes. The cognitive readiness measure consisted of (a) problem solving 
questionnaire (b) teamwork questionnaire, and (c) creativity questionnaire. 

Cognitive Readiness Measure 

The cognitive readiness measure was composed of a problem solving 
questionnaire, teamwork skills questionnaire, and creativity questionnaire. The problem 
solving questionnaire was composed of three scales from the Student approaches to 
learning (SAL) questionnaire developed by the Organization for Economic Co-operation 
and Development (OECD, Artelt, 2005; Baumert et al., 1998; Baumert, Klieme et. al., 
2000; Peschar, 2004). The reliability and validity of these three measures in the SAL 
questionnaire have been analyzed and validated by the cross national study conducted by 
Marsh et al. (2006). The self-regulation questionnaire also included a test anxiety scale 
adopted from the Test Anxiety Inventory (Speilberger, Gonzalez, & Taylor, 1980). 

The following scales were included in the problem solving questionnaire for this 
study: Self-efficacy, control strategies, elaboration, effort and perseverance, and test 



anxiety. 
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Teamwork Skills Questionnaire 

The teamwork questionnaire had six subscales. The subscales were: coordination, 
decision making, leadership, interpersonal skills, adaptability, and communication. There 
were total of 36 items in this questionnaire. The older version of the teamwork 
questionnaire was given to students. The fifty-four students were required to read the 
following directions to answer the questions. This set of questions is to help us 
understand the way you think and feel about working with others. Read each statement 
below and indicate how you generally think or feel. Indicate by marking an X on the 
appropriate box. There is no right or wrong answers. Do not spend too much time on each 
question. Item responses ranged from (1) almost never, to (2) sometimes, (3) often, and 
(4) almost always. The reliability reported by O’Neil (2003) for the version used in this 



study ranged from .84 to .90. 
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Creativity Questionnaire 

The creativity measure was adopted from Abedi and Schumacher (2002), this 
questionnaire consisted of 60 items, multiple choice test. Fifty- four students answered 
this questionnaire. 

Procedure 



Figure 4: Pilot Study Flow Chart 




The pilot study began with the researcher introducing the measures and providing 
a pre-test to the students and later a post-test. Participants in the study first filled out a 
problem solving questionnaire, teamwork questionnaire, and creativity questionnaire. 

In this study, the researcher modified previous researcher’s problem-solving 
measures of retention and transfer (Mayer 2004, Chen 2005) to measure domain specific 
problem solving strategies. The researcher measured participants’ problem-solving 



strategies with a set of retention and transfer questions. These problem-solving strategy 
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questions of retention and transfer are modified from Mayer and Moreno’s (1998) 
problem solving question list which were both used in the pilot and main study. 

Debriefing 

During the debriefing, the investigator thanked the participants for their time and 
effort. The investigator asked the participants to write down their suggestions or feelings 
on the debriefing sheet. The investigator also prompted a discussion about their feedback 
in terms of the assessments and overall experience. 

Results of the Pilot Study 

The research method and procedure of the pilot study were feasible, with the 
South Coast Naval School specific measures being: problem solving questionnaire, 
teamwork questionnaire, and creativity questionnaire. Further the research content was 
selected from the Multi-Mission Team Trainer. In this study the researcher modified 
previous researchers’ problem solving measures of retention and transfer questions 
(open-ended). For this research, the retention question was “Write an explanation of how 
you solve the tactical problem scenario. Be as specific as to the tactical scenarios.” The 
transfer question stated “Write some ways of improving the scenario used in the Multi- 
Mission Team Trainer.” 

Main Study 

In the end of February 2008, the investigator conducted the main study. The main 



study was conducted with the Multi Mission Team Trainer and paper and pencil 
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questionnaires, just like the pilot study. However, due to the schedule of classes at 
South Coast Naval School the content of the class changed. The course for the main study 
concentrated on surface warfare scenarios instead of air defense. Moreover, for the main 
study students were also given a retention and transfer question to answer which 
pertained to the culminating surface warfare scenario. Three experts instead of two were 
also given the same scenario and retention and transfer questions just like the students. 
Participants in the main study were requested to take the cognitive readiness 
questionnaires before experiencing the Multi-Mission Team Trainer and after they were 
done with the surface warfare module. They filled out the cognitive measure that 
included problems solving, teamwork, and creativity questionnaires, the same as those in 
the pilot study. In addition to this study, one other study was conducted respectively by 
another investigator (Mr. Sutter Fox) at the same time (Sutter, in preparation). The study 
focused on the formative evaluation of a surface warfare simulator to improve problem 
solving. 

Method 

Participants 

There were fifty eight participants and three instructors in the main study. 
However, only fifty-four participants took the pre and post test for the cognitive readiness 
measures. The participants that were sampled were from the South Coast Naval School. 



The participants were enrolled in the surface warfare module. The three experts had 12, 
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10, and 7 years of experience respectively. There were a total of three surface warfare 
case scenarios that were given to the experts. The experts had to classify all three case 
scenarios from lowest cognitive readiness to highest cognitive readiness. The following 
was the surface warfare case scenario that was given to the experts: 

Table 5: Surface Warfare Case Scenarios 



Scenario 1: 


• Build and maintain an RMP 

• Recommend various organic helicopter weapons to 
accomplish mission. 

• Successfully employ the SUW weapon systems. 

• Execute OTH-T engagement 


Scenario 2: 


• Build and maintain a recognized maritime picture 

• Identify battle space management considerations 

• Implement, determine, and implement the appropriate 
MIO Operations procedures 

• Implement TAO immediate actions for Search and 
Rescue 


Scenario 3: 


• Build a recognized maritime picture 

• Maintain a recognized maritime picture 

• Manage their battle space 

• Recommend various organic helicopter weapons to 
accomplish mission 

• Successfully employ the optimal shipboard SUW 
weapons systems. 

• Execute OTH-T engagement 

• Conduct SAG Operations 



Table 6 below explains the levels of cognitive readiness that all three experts 
agreed upon. Since all three experts agreed that all three surface warfare case scenario 
required higher levels of cognitive readiness, the researcher asked them to please identify 





from lowest to highest. Therefore, the following chart explains how they categorized 
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the scenarios. Scenario one was considered to require the lowest level of cognitive 
readiness. Scenario two required an intermediate level of cognitive readiness. Scenario 
three required the highest level of cognitive readiness. 

Table 6: Surface Warfare Case Scenario-Cognitive Readiness 



Scenario 1: (requires lowest 
cognitive readiness) 


• Build and maintain an RMP 

• Recommend various organic helicopter 
weapons to accomplish mission. 

• Successfully employ the SUW weapon 
systems. 

• Execute OTH-T engagement 


Scenario 2: (intermediate 
cognitive readiness level ) 


• Build and maintain a recognized maritime 
picture 

• Identify battle space management 
considerations 

• Implement, determine, and implement the 
appropriate MIO Operations procedures 

• Implement TAO immediate actions for Search 
and Rescue 


Scenario 3: (requires highest 
cognitive readiness) 


• Build a recognized maritime picture 

• Maintain a recognized maritime picture 

• Manage their battle space 

• Recommend various organic helicopter 
weapons to accomplish mission 

• Successfully employ the optimal shipboard 
SUW weapons systems. 

• Execute OTH-T engagement 

• Conduct SAG Operations 
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Table 7 below describes and defines the abbreviations and acronyms that were used in the 
surface warfare case scenario. 

Table 7: Acronyms/Abbreviations 



Terms 


Definitions 


RMP 


Recognized Maritime Picture 


MIO 


Maritime Interception operations- Efforts to monitor, 
query, and board merchants international waters to 
enforce sanctions. 


TAO 


Tactical Action Officers- key person who makes 
decisions 


SUW 


Surface Warfare 


OTH-T 


Over the Horizon targeting 


SAG 


Surface Action Group 



The above case surface warfare case scenarios were obtained from the South 
Coast Naval School and created by the instructor (experts) who taught the surface 
warfare component (see appendix F). 

The goal of surface warfare case scenario three was to train the Tactical Action 
Officer and the Surface Warfare Commanders to communicate, build and maintain 
recognize maritime picture using organic sensors while operating in a surface action 
group, make proper track and conduct a detect to engage sequence in a wartime rules of 



engagement environment against multiple hostile over the horizon surface tracks. 
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Measures 

The measures are divided into two categories, domain specific and domain 
independent. The cognitive readiness questionnaire (domain independent) included a 
problem-solving, teamwork, and creativity measure which was given to all fifty-four 
participants. 

Domain Specific Problem Solving Measure 

The following were the retention and transfer questions that were given to the 
students and experts pertaining to the surface warfare case scenario three: Write an 
explanation of how you solve the tactical problem scenario. Be specific as to the tactical 
scenarios. Write some ways of improving the scenario used in the Multi-Mission Team 
Trainer. The questions were related to the surface warfare case scenario three, which 
required the highest level of cognitive readiness. There were two versions of the retention 
and transfer questions. One version was given during the pilot study only to the two 
experts that were interviewed. After obtaining feedback, the retention and transfer 
questions were refined and given to the students during the post test (main study). 

The goal was to compare the responses between the expert and student learners. 
The investigator and the retired naval captain (Mr. Sutter Fox) thoroughly reviewed each 
response from the experts and from the students. The idea units were based both on the 
performance objectives and responses of the students. The responses from the experts 




were created into item units and compared to the responses of the students. The 
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responses of the students were also converted into idea units. The total number of idea 
units for the retention and transfer scores were ten. Therefore, the total number of points 
possible for students to obtain for each retention and transfer question were ten. 

They were given three scenarios to rank from highest to lowest cognitive 
readiness. The directions stated: How would you categorize the following Multi-Mission 
Team Trainer scenarios as requiring cognitive readiness? The likert scale consisted of 
four choices ranking from 1 (almost never), 2 (sometimes), 3 (often), and 4 (almost 
always). The pilot study consisted of the scenario being air defense. However, due to the 
course and timing of the main study the scenario changed to surface warfare. 



The three experts ranked the Surface Warfare case scenarios as follows: 
Table 8: Response of scenario ranking 





Scenario 1 


Scenario 2 


Scenario 3 


Expert 1 


3 


4 


4 


Expert 2 


4 


3 


4 


Expert 3 


4 


4 


4 
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As seen on table 9 (below), all three experts agreed that all scenarios required 
higher levels of cognitive readiness. After the three experts ranked the scenarios the 
investigator still wanted to know overall which one in their opinion was the one that 
required the least and highest level of cognitive readiness. The experts all agreed that 
overall scenario one was the one that required the least and scenario three required the 
highest level of cognitive readiness. 



Table 9: Overall scenario ranking from highest to lowest 





Scenario 1 


Scenario 2 


Scenario 3 


Expert 1 


Least 


Intermediate 


Highest 


Expert 2 


Least 


Intermediate 


Highest 


Expert 3 


Least 


Intermediate 


Highest 



Moreover, the open-ended questions (retention and transfer) were given to 
students in order to analyze how well they performed in comparison to the experts. The 
interrater reliability was conducted by the investigator and a retired naval captain. The 
experiences of the retired naval captain include thirty-four years of military service. His 
operational assignments included service aboard five ships, six air stations, and two 
group offices. His administrative experience includes the service headquarters in 
Washington DC, and US Pacific Command. 





For the retention and transfer questions both raters gathered and created an item 
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category from the answers that were provided by the experts. For example, after both 
raters created their own list they discussed and combined items into one single list, which 
included all three expert responses. After, thorough review the raters read each of the 
student’s responses for retention and mapped the items to that of the experts, as may be 
seen in table 10 below. There were a total of ten idea units for the retention expert 
categories. The ten idea units were the following: 1. communication, 2. 
coordinate/organize, 3. recognize maritime picture (RMP), 4. Search Plan, 5. Identify and 
classify contacts, 6. contact, 7. visual identification, 8. maintain surprise/ emissions 
control (EMCON), 9. best weapons, and 10. battle damage assessment (BDA). The same 
procedure was conducted for the transfer question. The idea units were also aligned to the 
objectives of the three surface case scenarios (see appendix). 




Table 10: Cross Reference Between Performance Objectives and Idea Units 
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Performance 

objectives 




1 . 


2. 


3. 


4. 


5. 


6. 


7 


8. 


9 10. 


1 . 




X 
















2. 






X 


X 


X 




X 






3. 




X 


X 


X 


X 


X 


X 






4. 


















X 


5. 


















X 


6. 












X 


X 






7. 


X 


X 












X 





The marks on table 10 (above) indicated the performance objectives that were 
mapped onto the idea units. The performance objectives were the following: 1. manage 
their battlespace, 2. build a recognized maritime picture, 3. maintain a recognized 
maritime picture, 4. recommend various organic helicopter weapons to accomplish 
mission, 5. successfully employ the optimal shipboard SUW weapon system, 6. execute 
over the horizon targeting engagement, and 7. conduct surface action group operations. 
Every performance objective, mapped onto one or more of the idea units. Likewise, 
every idea unit, mapped onto at least one of the performance objectives. The exception 
was battle damage assessment idea unit, which is one idea unit that emerged from 
participant’s responses and was not in the performance objectives. 

The two raters analyzed one by one the responses of the students and mapped the 



answers to those of the students. Each student was given one point when the two items 
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matched. The total number of transfer units that were derived from the three expert 
responses consisted of ten, as see in table 11 below. 

Table 11: Item Units for Transfer question 



1. Increase scenario Fidelity 

2. Improve link modeling 

3. Improve Tactical aircraft employment 

4. Improve reporting procedures 

5. Add ability to organize assets other than own ship 

6. Improve gunfire simulation 

7. Improve environment modeling 

8. Improve missile attack coordination 

9. Improve optical sight 

10. Improve bridge/CIC coordination 



Domain Independent Problem Solving Measure 
Problem Solving Measure 

The domain independent problem solving questionnaire was composed of a subset 
of twenty-three total items. The subscales included were: control strategies 
(metacognition), academic efficacy (self-efficacy), effort and perseverance and test 
anxiety (worry component), and elaboration. The details of each subscale of the trait self- 
regulation questionnaire are as follows: 

Metacognition (control strategies) describes students that used self-evaluation and 
self-monitoring during their learning process (Marsh et. al., 2006). A four-point Likert 



scale, ranging from almost never, sometimes, often, to almost always, was used to 
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measure metacognition. The reliability of metacognition was .83 (Marsh et al., 2006). 

A sample question is “When I study, I force myself to check to see if I remember what I 
have learned” (Marsh et al., 2006). 

Academic efficacy (Self-Efficacy) is the “people’s judgment of their capabilities 
to organize and execute course of action to attain designated types of performance.” Self- 
efficacy also refers to the one’s belief in his or her ability to complete a difficult task to 
overcome barriers to reach the goal successfully through efforts (Marsh et.al., 2006). A 
four point Likert scale ranging from almost never, sometimes, often, to almost always 
was used to measure this subscale. The reliability of the academic self-efficacy measure 
was .83 (Marsh et. al., 2006). A sample question to measure self-efficacy was: “I am 
confident I can understand the most complex material presented by the teacher” (Marsh 
et al., 2006). 

Effort and perseverance. Learners spend more effort when they are motivated. 
Effort and perseverance are positively related to students’ achievement (Pintrich & 
Schunk, 2002). A four-point Likert scale, ranging from almost never, sometimes, often, 
to almost always, will be used to measure effort. The reliability of the effort measure is 
.83 (Marsh et. al., 2006). A sample question for this subscale is “When studying, I keep 
working even if the material is difficult” (Marsh et al. 2006). 

Test anxiety (worry). The test anxiety inventory (TAI) is a self-report inventory 
that measures two components worry and emotionality. Worry relates to the cognitive 
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concerns about the consequences of failure. Emotionality refers to the reactions of the 
automatic nervous system that is evoked by evaluative stress (Speilberger et al., 1980). 
The reliability of the test anxiety measure is .92 (Speilberger et al., 1980). A sample 
question to measure the worry component of test anxiety was “Thoughts of doing poorly 
interfere with my concentration on tests” (Speilberger et al., 1980). 

Teamwork Measures 

The main study used the same teamwork measure questionnaire as the pilot study. 
There were a total of thirty-six questions. The measurement approach is to view 
teamwork skills as a trait of the individual that predisposes the individual to act as a team 
member (O’Neil, 2003). The taxonomy is made up of six teamwork processes. The skill 
include (a) adaptability- recognizing problems and responding appropriately, (b) 
coordination- organizing team activities to complete tasks on time, (c) decision making- 
using available information to make decisions, (d) interpersonal- interacting 
cooperatively with other team members; (e) leadership- providing direction for the team; 
and (f) communication- clear and accurate exchange of information. 

An example of the adaptability subscales is: “When I work as part of a team, I can 
identify potential problems readily.” An example of the coordination subscales is: “When 
I work as part of a team. I allocate the tasks according to each team member’s abilities.” 
An example of example of decision-making is: “When I work as part of a team, I 
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understand and contribute to organizational goals.” An example of interpersonal is: 
‘‘When I work as part of a team, I interact cooperatively with other team members.” An 
example of leadership is: “When I work as part of a team, I exercise leadership.” An 
example of communication is:” When I work as part of a team, I ensure the instructions 
are understood by all team members prior to starting the task” (Spector, Ohrazda, 
Schaack, & Wiley, 2005, pg. 146). 

The item responses ranged from (1) almost never, to (2) sometimes, (3) often, to 
(4) almost always. The reliability reported by O’Neil (2003) for the version used in this 
study ranged from .84 to .90. 

Creativity Measures 

The creativity questionnaire was composed of sixty multiple choice items and was 
adopted from Abedi and Schumacher (2002). There were four subscales. The subscales 
consisted of fluency, flexibility, originality, and elaboration. According to Abedi & 
Schumacher (2002) who adopted the definitions from Torrance & Goff (1989) fluency 
refers to ability to produce large numbers of ideas, flexibility (ability to produce a variety 
of ideas or use a variety of approaches), originality (ability to produce new, unusual, 
innovative ideas), and elaboration (ability to fill in the details). An example of the 
fluency question is: “How well do you express your ideas?” An example the flexibility 
question is “How do you approach a complex task?” An example of the originality 



question is “Do you find that you usually like to try new things?” An example of the 
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elaboration question is “When something unfair happens to you, can you figure out all 
of the reasons behind that event?” 

The directions given to the students were the following: This set of questions is 
to help us understand your level of creativity in different situations. Read each statement 
below and bubble in one answer per question. Do no spend too much time on each 
question. The multiple-choice test is based on the original version of a creativity test 
constructed and validated in Tehran by Abedi in 1983. Reliability and validity were 
examined in a study conducted in Spain on a group of 2,270 students (Villa, Auzmendi, 
& Abedi, 1996). 

Each of these measures of creativity yields four different subscales: Fluency, 
Flexibility, Originality, and Elaboration. Fluency has 22 items, Flexibility has 11 items, 
Originality has 16 items, and Elaboration has 11 items. Each item has three options 
(choices) ranging from the least to the most creative response. Internal consistency 
coefficients of the creativity test were estimated using the classical Cronbach alpha. 
Concurrent validity coefficients were estimated by correlating the scores of the creativity 
test subscales with the criterion measures. Internal consistency and concurrent validity 
coefficients were also estimated by utilizing multiple indicators through the application 



of structural equation modeling. 
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Figure 5: Main Study Flow Chart 
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The main study began with the researcher introducing the measures and providing 
a pre-test to the students and later a post-test. Participants in the study first filed out a 
problem solving questionnaire, teamwork questionnaire, and creativity questionnaire. 
After, all participants answered the retention and transfer questions. The retention and 
transfer questions were also answered by three experts. The experts consisted of three 
instructors who had been teaching the surface warfare 12, 10, and 7 years respectively. 
Debriefing of the Main Study 

The process of debriefing after the main study was the same of the pilot study. 

The investigator first thanked the participants for their time and patience in the study. The 
investigator asked participants to write down their suggestions or feelings toward the 



process they had undergone. 
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CHAPTER IV 
RESULTS 

Data collected in this study was analyzed by using the Statistical Package for the 
Social Sciences (2006). The statistical procedure included both descriptive and inferential 
statistics. The scale of problem solving was analyzed first (subscales: self- efficacy, 
elaboration, worry, anxiety, control strategies, effort and perseverance). Then the 
teamwork questionnaire was analyzed, the subscales included cognition and 
interpersonal. Next, the subscales of the creativity measure were presented. The subscales 
were flexibility, fluency, elaboration, and originality. The pre and post test were analyzed 
via Pair T-Test and correlations. Last, the interrater reliability of the retention and 
transfer questions were analyzed. The results based on the research question and one 
hypothesis were summarized. The numbers were truncated to two significant digits. 

Descriptive and Inferential Statistics 

Table 12 shows the mean and standard deviations for all subscales. For all 
subscales the mean for pre and post test were approximately the same. 




Table 12: Subscales: Items, Means, and Standard Deviation 



Subscales 



Pre CS 



Post CS 



Pre SE 



Post SE 



Pre E & P 
Post E & P 
Pre Worry 
Post Worry 
Pre P-Elab 
Post P-Elab 
Pre Adapt 
Post Adapt 
Pre Coord. 
Post Coord. 
Pre DM 
Post DM 
Pre Interper 
Post Interper 
Pre Lead 
Post Lead 
Pre Comm 
POST Comm 
Pre C-Flu 
Post C-Flu 
Pre C-Flex 
Post C-Flex 
Pre C-Orig 
Post C-Orig 
Pre C-Elab 
Post C-Elab 
Ret. Score 
Tran- Score 



No. of Items 



Std. Deviation 



15.68 




15.55 




12.55 









12.55 


2.44 


12.75 


2.47 


10.53 


4.46 


10.57 


4.35 


9.38 


1.74 


9.40 


1.80 


16.15 


2.52 


16.91 


2.12 


15.83 


2.23 


16.23 


2.14 


19.38 


2.80 


19.81 


2.72 


16.06 


1.85 


16.85 


1.87 


22.60 


2.56 


23.23 


2.41 


22.06 


2.56 


22.45 


2.55 


37.92 


6.00 


38.13 


6.05 


28.30 


5.16 


28.72 


3.95 


37.55 


5.69 


38.30 


4.32 


22.74 


3.33 


23.45 


2.56 


2.38 


2.24 


.94 


1.15 




















Table 13 shows the alpha reliability of the pre and post test scores for all 
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subscales and the retention and transfer question. The alpha reliability for all subscales 
ranged from .50 to .93. 

Table 13: Alpha Reliability for all Subscales 



Subscales 


Alpha 

Reliability 

(Pre-test) 


Alpha 

Reliability 

(Post-Test) 


Alpha Reliability 
(Reference) 


Control Strategies 


.68 


.80 


.83 (Marsh, et. al. 2006) 


Self-Efficacy 


.87 


.82 


.83 (Marsh, et. al. 2006) 


Effort and 
Perseverance 


.81 


.85 


83 (Marsh, et. al. 2006) 


Worry 


.92 


.93 


.92 (Marsh, et. al, 2006) 


Problem Solving- 
Elaboration 


.72 


.69 


.83 (Marsh, et. al. 2006) 


Teamwork- 

Adaptability 


.69 


.78 


.84 (O’Neil, 2003) 


Teamwork- 

Coordination 


.60 


.66 


.84 (O’Neil, 2003) 


Teamwork- 
Decision Making 


.75 


.78 


.84 (O’Neil, 2003) 


Teamwork- 

Interpersonal 


.56 


.69 


.84 (O’Neil, 2003) 


Teamwork- 

Leadership 


.48 


.55 


.84 (O’Neil, 2003) 


Teamwork- 

Communication 


.57 


.65 


.84 (O’Neil, 2003) 


Creativity- 

Fluency 


.86 


.74 


.79 (Abedi,2002) 


Creativity- 

Flexibility 


.75 


.85 


.53 (Abedi,2002) 


Creativity- 

Originality 


.76 


.83 


.64 (Abedi,2002) 


Creativity- 

Elaboration 


.50 


.79 


.64 (Abedi,2002) 
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Problem Solving Measure 

Problem Solving was assessed through a questionnaire that was based adopted 
from the Student Approaches to Learning (SAL), which was developed by the 
Organization for Economic Co-Operation and Development (OECD, Artelt, 2005; 
Baumert et al., 1998; Baumert, Klieme et. al., 2000; Peschar, 2004). The questionnaire 
consisted of twenty-five items and was a four- point Likert scale, ranging from almost 
never, sometimes, often, to almost always. The subscales included self-efficacy (4 items), 
elaboration (4 items) , test anxiety (worry) (8 items) , control strategies (5 items), and 
effort and perseverance (4 items). Each subscale will be analyzed first descriptively and 
then inferentially. The hypothesis stated that there would be an increase of scores from 
the pre to post test. 

Control strategies 

Control Strategies was measured via the control strategies subscale, cognitive readiness 
questionnaire. Table 14 below, shows the mean and standard deviation for the pre and 
post control strategies subscale. The mean for the pretest and post test was (15.68) and 
(15.55). The means were approximately the same. The independent variable was the time 
of which the constructs were measured, ie.. before and after the surface warfare 




70 

simulation. The T-test shows that there in no statistical significance between the pre 
and post test control strategies subscale ( t=.44, df= 53, p=.65). 



Table 14: Control Strategies- Mean and Standard Deviation (N=54) 





Mean 


Std. Deviation 


Pre Control 
Strategies 


15.68 


2.35 


Post Control 
Strategies 


15.55 


2.57 



Self-efficacy 

Self-efficacy was measured via the self-efficacy cognitive readiness 
questionnaire. Table 15 below, shows the mean and standard deviation for the self- 
efficacy pre and post test. The mean score for the pre-test was 12.55 and for the post-test 
12.96. 

A pair sample T-Test was conducted on each subscale to determine if there was 
any difference and significance between the pre test and post test. The independent 
variable was the time of which the constructs were measured, ie.. before and after the 
surface warfare simulation. The pair T-Test shows that there was no statistical 



significance, (t=1.4, df=53, p=.15). 





Table 15: Self-Efficacy -Mean and Standard Deviation (N=54) 
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Mean 


Std. Deviation 


Pre Self- Efficacy 




12.55 


2.85 


Post Self- Efficacy 




12.96 


2.51 



Effort and perseverance 

Effort and perseverance was measured via the effort and perseverance cognitive 
readiness questionnaire. Table 16, below depicts the mean and standard deviation. The 
mean was 12.51 for the pre test and 12.75 for the post test, being approximately the same. 
The independent variable was the time of which the constructs were measured, ie.. before 
and after the surface warfare simulation. The pair t-test showed no statistical significance, 
(t=.90, df= 53, p=. 33). 



Table 16: Effort and Perseverance- Mean and Standard Deviation (N=54) 





Mean 


Std. Deviation 


Pre Effort & Pers. 


12.51 


2.44 


Post Effort & Pers. 








12.75 


2.47 



Worry 

Worry scales was measured via the worry subscale cognitive readiness questionnaire. 
Table 17 below, depicts the mean and standard deviation. The mean for pre test was 
(10.53) and the mean for the post test was (10.57). The mean was approximately the 
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same. The independent variable was the time of which the constructs were measured, 
ie.. before and after the surface warfare simulation. The pair T-Test showed that there is 
no statistical difference (t=.06, df =53, p=.94). 



Table 17- Worry- Mean and Standard Deviation (N=54) 





Mean 


Std. Deviation 


Pre Worry 


10.53 


4.46 


Post Worry 


10.57 


4.35 



Elaboration 

Elaboration was measured via elaboration subscale cognitive readiness 
questionnaire. Table 18 below, shows the means and standard deviation between the pre 
and post test. The mean for the pre test was 9.38 and the mean for the post elaboration 
was 9.40. The independent variable was the time of which the constructs were measured, 
ie.. before and after the surface warfare simulation. The pair sample T-Test demonstrated 
that there was no statistical significance between the pre and post test (9.3, 9.4), (t=. 10, 
df=53, p= .91). 

Table 18: Elaboration -Mean and Standard Deviation (N=54) 





Mean 


Std. Deviation 


Pre P Elab 


9.38 


1.74 


Post P Elab 


9.40 


1.80 





Teamwork Measure 
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Teamwork was assessed via a cognitive readiness questionnaire that was given 
before and after the simulation. There were thirty-six items in this questionnaire, which 
consisted of six scales (adaptability-5 items, coordination- 5 items, decision making-6 
items, interpersonal-6 items, leadership- 7 items, and communication-7items). Moreover, 
each subscale will show the mean, standard deviation, and results for the paired T-Test. 

Teamwork- adaptability was measured via O’Neil et. al. teamwork questionnaire. 
Table 19 below, shows that Teamwork- adaptability mean and standard deviation. The 
mean was pre-test (16.15) and the post-test (16.91). The independent variable was the 
time of which the constructs were measured, ie. before and after the surface warfare 
simulation. The pair T-Test showed it was statistically significant, (t= 2.6, df=53, p=.01). 
Table 19: Teamwork- Adaptability- Mean and Standard Deviation 





Mean 


Std. Deviation 


Pre- Adapt 


16.15 


2.52 


Post- Adapt 


16.91 


2.12 



Teamwork- coordination was measured via the O’Neil et. al. teamwork 
questionnaire. Table 20 below, describes the mean and standard deviation for the pre and 
post test for coordination. The means are approximately the same, pre test (15.83) and 
post test (16.23). The independent variable was the time of which the constructs were 





measured, ie.. before and after the surface warfare simulation. The paired T-Test 
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showed that there is no statistical significance, (t= 1.4, df= 53, p= .16). 



Table 20- Teamwork- Coordination- Mean and Standard Deviation 





Mean 


Std. Deviation 


Pre-Coordination 








15.83 


2.23 


Post- Coordination 








16.23 


2.14 



Teamwork-decision making was measured via O’Neil et. al teamwork questionnaire. 
Table 21 below, shows the mean and standard deviation for pre test (19.38) and post test 
(19.81) for decision making. The independent variable was the time of which the 
constructs were measured, ie.. before and after the surface warfare simulation. The pair 
T-test shows that Teamwork-decision has no statistical significance, (t= 1.2, df= 53, 

p=.21). 

Table 21: TW-Decision Making- Mean and Standard Deviation (N=54) 





Mean 


Std. Deviation 


Pre- Decision Making 


19.38 


2.80 


Post- Decision Making 


19.81 


2.72 








Teamwork-interpersonal was measured via O’Neil et. al. teamwork. Table 22 
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below, shows the mean and standard deviation for Teamwork- Interpersonal. The pretest 
mean was (16.06) and the post test mean was (16.85). The independent variable was the 
time of which the constructs were measured, ie.. before and after the surface warfare 
simulation. The pair T-Test showed that there is statistical significance, (t=3.1, df=53, 

p=.00). 

Table 22: Teamwork- Interpersonal (N=54) 





Mean 


Std. Deviation 


Pre-Interpersonal 


16.06 


1.87 


Post- Interpersonal 


16.85 


1.85 
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Teamwork- leadership was measured via O’Neil et. al teamwork questionnaire. Table 23 
below, shows the mean and standard deviation for Teamwork-. The 
Mean for the pre test was (22.60) and the mean for the post test was (23.23). The 
independent variable was the time of which the constructs were measured, ie.. before and 
after the surface warfare simulation. The pair T-test showed that there is a statistical 
significance, (t= 2.2, df = 53, p=.02). 

Table 23: Teamwork- Leadership- Mean and Standard Deviation (N=54) 





Mean 


Std. Deviation 


Pre- Leadership 


22.60 


2.55 


Post- Leadership 


23.23 


2.41 



Teamwork- communication was measured via the O’Neil et. al. teamwork questionnaire. 
Table 24 shows the mean and standard deviation for Teamwork- communication. The mean 
was 22.06 (pre-test) to 22.49 (post-test). The independent variable was the time of which 
the constructs were measured, ie.. before and after the surface warfare simulation. The pair 
T-test showed that the results were not statistically significant, (t= 1.3, df= 53, p=.18). 









Table 24-Teamwork- Communication- Mean and Standard Deviation 
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Mean 


Std. Deviation 


Pre- Communication 












22.06 


2.56 


Post- Communication 












22.49 


2.55 



Creativity Measure 

Creativity was assessed through a questionnaire that was given before and after 
the simulation. The creativity questionnaire had four subscales, which included, 
flexibility (13 items), elaboration (10 items), originality (16 items), and fluency (17 
items). The creativity measured was given before and after the surface warfare 
simulation. The tables below will show the mean, standard deviation, and T-Test to 
determine if the results were significant. 

Fluency 

Creativity- fluency was measured via Abedi’s creativity questionnaire. Table 25 below 
shows the mean and standard deviation for fluency. The fluency mean went from 37.92 
(pre-test) to 38.13 (post-test). The independent variable was the time of which the 
constructs were measured, ie.. before and after the surface warfare simulation. The pair 
T-Test shows that there is no statistical significance, (t= .28, df=53, p=.78). 
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Table 25: Creativity: Fluency- Mean and Standard Deviation (N=54) 





Mean 


Std. Deviation 


Pre Fluency 


37.92 


6.00 


Post Fluency 


38.13 


6.05 



Flexibility 

Creativity- flexibility was measured via Abedi’s creativity questionnaire. Table 26 
below, shows the mean and standard deviation for pre and post test. The mean for the pre 
test is 28.30 and 28.72 for the post test. The independent variable was the time of which 
the constructs were measured, ie.. before and after the surface warfare simulation. The 
pair T-Test showed that there is no statistical significance, (t= .90, df= 53, p= .37). 



Table 26: Creativity- Flexibility- Mean and Standard Deviation (N=54) 





Mean 


Std. Deviation 


Pre- Flexibility 


28.30 


5.16 


Post- Flexibility 


28.72 


3.95 
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Originality 

Creativity- originality was measured via Abedi’s creativity questionnaire. Table 
27 below, shows the mean and standard deviation for the originality subscale. The mean 
for the pre test was 37.55 and the mean for the post test was 38.30. The independent 
variable was the time of which the constructs were measured, ie.. before and after the 
surface warfare simulation. The pair T-test showed that there is no statistical significance, 
(t=1.4, df= 53, p= .16). 

Table 27: Creativity- Originality -Mean and Standard Deviation (N=54) 





Mean 


Std. Deviation 


Pre Originality 


37.55 




5.69 


Post Originality 


38.30 




4.32 



Elaboration 



Creativity- elaboration was measured through Abedi’s creativity questionnaire. 
Table 28 below shows the mean for the pre elaboration score was 22.74 and the mean for 
the post elaboration score was 23.45. The independent variable was the time of which the 
constructs were measured, ie.. before and after the surface warfare simulation. The pair 



T-Test showed it approached significance, (t=1.9, df=53, p=.05). 
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Table 28: Creativity- Elaboration- Mean and Standard Deviation (N=54) 





Mean 


Std. Deviation 


Pre C- Elab 








22.74 


3.33 


Post C- Elab 








23.45 


2.56 



In order to investigate the test-re test reliability of the measured used in the study 
the pre test was correlated with the post test. As may be seen below, the correlation 
ranges from .53 to .75. 





Correlation Matrix 



Table 29: Correlations- Pre and Post Test 



Subscales 


No. of Items 


Pre-Test / Post-Test 


Control Strategies 


5 


.61** 


Self-Efficacy 


4 


.69** 


Effort and Perseverance 


4 


72** 


Worry 


8 


.53** 


Elaboration 


4 


71 ** 


Adaptability 


5 


.61** 


Coordination 


5 


.54** 


Decision Making 


6 


.57** 


Interpersonal 


6 


49 ** 


Leadership 


7 


. 66 ** 


Communication 


7 


.56** 


Fluency 


17 


.59** 


Flexibility 


13 


.75** 


Originality 


16 


72** 


Creativity Elaboration 


10 


.62** 


Retention/Transfer 


10 each 


.12 



**: Correlation is significant at the 0.01 level (2-tailed). 





In order to test the relationship between the sub scales and retention and 
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transfer measure, a correlation between subscales was conducted. The pre test variable 
was correlated with the retention and transfer measure. The correlations amongst the 
pretest and retention ranged from -.02 to .14. Only the correlation between self-efficacy 
and retention was significant. The correlations amongst the pretest and transfer ranged 
from -.03 to .10. There was no relation on transfer for the cognition retention measures. 




Table 30: Correlations- Pre-Test-Retention and Transfer 



Subscales - Pre- Test 


Retention 


Transfer 


Control Strategies 


.14 


.00 


Self-Efficacy 


.27* 


-.12 


Effort and Perseverance 


.06 


.10 


Worry 


-.23 


.09 


Elaboration 


.00 


-.09 


Adaptability 


.00 


.02 


Coordination 


-.16 


-.05 


Decision Making 


-.13 


.06 


Interpersonal 


-.00 


-.03 


Leadership 


-.17 


-.07 


Communication 


-.12 


.02 


Fluency 


-.02 


-.08 


Flexibility 


.05 


-.20 


Originality 


-.03 


.-15 


Creativity Elaboration 


.15 


-.08 


Transfer 


.12 





* Correlation is significant at the 0.05 level (2-tailed). 





In order to test the reliability between the subscales (post-test) and retention 
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and transfer measure, correlations were conducted. The post test measure was correlated 
amongst the retention and transfer. The correlations for the post test and retention 
measure range from -.06 to .26. 




Table 31: Post-Test- Retention and Transfer 



Subscales - 
Post-Test 


Retention 


Transfer 


Control Strategies 


.09 


-.19 


Self-Efficacy 


.26 


-.17 


Effort and Perseverance 


.11 


-.01 


Worry 


-.14 


.08 


Elaboration 


.08 


-.19 


Adaptability 


-.07 


-.07 


Coordination 


-.06 


-.11 


Decision Making 


-.10 


-.11 


Interpersonal 


.03 


-.11 


Leadership 


.03 


-.04 


Communication 


-.11 


-.08 


Fluency 


.07 


-.29* 


Flexibility 


.02 


-.34* 


Originality 


.04 


-.24 


Creativity Elaboration 


.17 


-.25 



* Correlation is significant at the 0.05 level (2-tailed). 
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Interrater Reliability of Domain Specific Problem Solving Measurement 

Interrater reliability is widely used term for the extent to which independent 
coders evaluate a characteristic of a message or artifact and reach the same conclusion. It 
measures only “the extent to which the different judges tend to assign exactly the same 
rating to each object “(Tinsley & Weiss, 2000, p. 28). A score for a retention question or 
a transfer question was the total number of the idea units in the student’s answer that 
matched the experts’ idea units. The experts list for the retention question included ten 
idea units. The experts list for the transfer questions included ten idea units. The total 
possible score for each retention and transfer questions was ten. There were a total of 
fifty-two students and three experts. 

The retired naval captain and the researcher first read all the responses provided 
by the three experts. After each rater read each response individually both agreed upon 
writing two separate sets of idea units. The result was ten idea units for the retention 
question and ten idea units for the transfer question. After having the two sets of idea 
units it was compared to that of the students. 

Table 32 below, shows the frequency of the retention scores of students. The 
largest percent (27.8%) of students received a two as a score out of ten total possible 
points. The least number of students (1.9%) received an eight out ten total possible 



points. It is interesting to note that 24% of received a score of zero. 
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Table 32 : Frequency of Retention Scores - Students (N=52) 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid 0 


13 


24.1 


24.1 


24.1 


1 


8 


14.8 


14.8 


38.9 


2 


15 


27.8 


27.8 


66.7 


3 


4 


7.4 


7.4 


75.9 


4 


2 


3.7 


3.7 


79.6 


5 


3 


5.6 


5.6 


85.2 


6 


4 


7.4 


7.4 


92.6 


7 


3 


5.6 


5.6 


98.1 


8 


1 


1.9 


1.9 


100.0 


Total 


52 


100.0 


100.0 





Table 33 below shows the frequency of the transfer scores of students. The 
largest percentage (44.4%) received the score of zero out of ten possible points. The 
lowest percentage (1.9) received the score of 6 out of ten possible points. 



Table 33: Frequency of Transfer Scores - Students (N=52) 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid 


0 


24 


44.4 


44.4 


44.4 




1 


1 


1.9 


1.9 


46.3 




2 


15 


27.8 


27.8 


74.1 




3 


10 


18.5 


18.5 


92.6 




4 


3 


5.6 


5.6 


98.1 




6 


1 


1.9 


1.9 


100.0 




Total 


52 


100.0 


100.0 
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Table 34 (see appendix G) shows that for each answer the student provided, it 
was matched with the expert. If the responses were the same students received a point. 
The total points possible were ten. The average score was 2.42, standard deviation, 2.26. 

Table 35 (see appendix H) shows the scores that the two raters gave each student 
for the retention question. After a thorough discussion between the two raters a consensus 
was achieved in order to finalize with one score. The Kappa analysis was conducted in 
order to analyze the interjudge reliability, .93. Table 36 (see appendix I) reflected the 
scores for the transfer question. The idea units from the three experts were mapped to that 
of the students. The average score for the transfer was .096 and standard deviation 1.17. 
Table 37 (see appendix J) shows the scores that the two raters gave students for the 
transfer questions. After a thorough discussion the rater came to a consensus to obtain 
one score. Kappa analysis was conducted in order to obtain the interjudge reliability. The 
reliability was .93. 

Tests of the Research Hypothesis 

Research Question One: 

Will a simulation increase cognitive readiness? 

Hypothesis 1: There will be an increase of scores between the pre and post cognitive 
readiness questionnaire. 

With the exception of teamwork- interpersonal and leadership, and creativity- 
elaboration, the T-test for each of the subscales showed no statistical significance. The 
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correlations that were statistically significant were self-efficacy and retention (.27) and 
creativity- fluency (-.29) and flexibility with transfer (-.34). 

The measures were divided into two categories domain independent (which 
included the problem solving, teamwork, and creativity) and domain specific (which 
included retention and transfer questions). It was hypothesized that post test scores were 
going to be higher than the pre test scores. The mean for each subscale (pre and post test) 
showed that they were approximately the same and not significant. The domain 
independent problem solving measure was composed of five subscales. The subscales 
were: self-efficacy, elaboration, worry, control strategies, and effort and perseverance. 

For example, the mean for self-efficacy went from (12.55) to (12.96) 
approximately the same. The mean for control strategies went from 15.68 to 15.55, the 
mean for effort and perseverance went from 12.55 to 12.75, the mean for worry went 
from 10.53 to 10.57, and the mean for elaboration went from 9.38 to 9.40. None of the 
means were significantly different. In addition, the relationship between the domain 
independent problem solving strategies and the retention and transfer questions indicated 
that there were no significant relationships. 

The alpha reliability for problem solving consisted of the following: control 
strategies (pre: .68, post: .80), self-efficacy (pre: .87, post: .82), effort and perseverance 



(pre: .81, post .85), worry (pre: .92, post: .93), elaboration (pre: .72, post: . 69). The alpha 
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reliability ranged from .68 to 93, which stated that most of the problem solving, 
domain independent measures were reliable, greater than .70. 

The teamwork measure consisted of six subscales. The six subscales were 
coordination, communication, decision-making, interpersonal, leadership, and 
adaptability. Two subscales were statistically significant. Teamwork interpersonal and 
leadership were statistical significant. The mean for Teamwork interpersonal went from 
16.06 to 16.85. The mean for Teamwork- Leadership went from 22.60 to 23.23. The 
mean for the subscales that were not statistically significant are adaptability (16.15, 

16.91), coordination (15.83, 16.23), decision making (19.38, 19.81), and communication 
(22.06, 22.45). The measure for interpersonal and leadership subscales were statistically 
significant. Thus, interpersonal dealt with interacting with other team members and 
leadership dealt with providing direction for the team (Marshal et. al, 2005) which 
increased after exposure to the Multi Mission Team Trainer. Unexpectedly, teamwork 
decision making was not significant. 

The alpha reliability for teamwork consisted of the following: adaptability (pre: 

. 69, post: .78), coordination (pre: .60, post: .66), decision making (pre:75, post:.78), 
interpersonal (pre: .56, post: 69), leadership (pre:. 48, post: .55), communication (pre: .57, 
post: .79). The reliability ranged from .48 to .79. Some of the measures showed to be 



below .70 and thus may need revision. 
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The creativity subscales included flexibility, elaboration, originality, and 
fluency. All of the subscales with the exception of creativity- elaboration were not 
statistically significant. The mean for creativity elaboration went from 22.74 to 23.45. 
Thus creativity-elaboration increased after exposure to the MMTT. Creativity-elaboration 
meant the abilities to fill in the details (Abedi & Schumacher, 2002), which approach 
statistical significance (P=.05). 

The means for the rest of the subscales were the following: originality (37.55, 
38.30), fluency (37.92, 38.13), and originality (37.55, 38.30). The alpha reliability for 
creativity consisted of the following: fluency (pre:.86, post:.74), flexibility (pre:.75, post: 
.85), originality (pre:.76, post: .83), and elaboration (pre: .50, post: .79). The reliability 
ranged from .50 to .86. All of the creativity subscales (with the exception of elaboration 
pre-test) showed to be reliable and greater than .70. 

The overall alpha reliability of all the subscales for the pre and post test ranged 
from .50 to .93. The subscales that were below .70 were control strategies (pre-test), 
problem solving (elaboration-post test), teamwork adaptability (pre-test), teamwork- 
coordination (pre and post), teamwork-interpersonal (pre and post), teamwork-leadership 
(pre and post), and creativity-elaboration (pre-test). These scales may also need revision. 

The correlations of all subscales with the retention question (pre-test) ranged from 
-.03 to .27, with self-efficacy being positively correlated with retention and showing 



statistical significance. The correlations with all subscales and the retention question 
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(post-test) ranged from -.06 to .17, showing no statistical significance. The correlations 
of all subscales and the transfer question (pre-test) ranged from -.08 to .00, showing no 
statistical significance. The correlations of all subscales and the transfer question (post- 
test) ranged from -.34 to .08, showing that fluency (-.29) and flexibility (.-34) were 
statistically significant but unexpectedly negative. It may be that SWOS education 
focuses on “school solutions” and does not reward fluency and flexibility. The 
correlation between retention and transfer was .12, showing no statistical significance. 

The correlations of all subscales (pre and post) ranged from .49 to .72. 

The domain specific problem solving strategies were tested via a modification of 
Mayer’s retention and transfer questions (1998). The retention question was: Write an 
explanation of how you solved tactical problems in the Multi Mission Team Trainer 
scenario you just completed? The transfer question was: Write an explanation of how 
you would improve the Multi Mission Team Trainer Scenario, that you just completed, to 
train for a complex and unpredictable warfare environment. 

The answers from students were compared to that of three experts from the South 
Coast Naval School (navy setting). The two raters (the researcher and a retired naval 
captain) scored each response provided by the fifty-four students and provided a score 
that ranged from one to ten. This process was performed twice, one for the retention 
question, and one for the transfer question. The two raters provided independently 
provided each student with a score, later the raters, agreed upon any discrepancies, and 
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reached a consensus. In order to find the interjudge reliability the researcher conducted 
a Kappa analysis. The kappa was .93 (after truncation) for both retention and transfer, 
which indicates a high reliability. 

In summary the hypothesis stated that there would be an increase of scores 
between the pre and post cognitive readiness questionnaire due to simulation 
intervention. This hypothesis was partially confirmed. The results consisted of teamwork- 
interpersonal and leadership and creativity-elaboration were statistically significant. 
Creativity- elaboration was approximately statistically significant, p=.05. Further some 
cognitive readiness attributes were significant, i.e. self-efficacy and retention (.27), and 
creativity flexibility (-.29) and fluency (-.34) and retention. The relationship with transfer 
was not unexpected given the data, i.e. the average performance was .096 compared to 
experts and 24 individuals out of 52 received a score of zero. 
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CHAPTER V 

Summary, Discussion, and Implications 

This chapter will focus on the summary, discussion, and implications of the study 
conducted. 

Summary 

This study examined the effects of cognitive readiness in a surface warfare 
simulation. The purpose of the study was to determine if student’s cognitive readiness 
increased from the pre test to the post test, the intervention being the Multi Mission Team 
Trainer simulation (a surface warfare case scenario). There was one group of participants 
which was composed of fifty four students. The intervention lasted seven days and the 
questionnaire approximately took 40 minutes for students to complete. The Multi Mission 
Team Trainer surface warfare simulation took approximately one and a half hours. There 
was a pre test composed of three different scales: problem solving (self-efficacy, 
elaboration, test anxiety, control strategies, and effort and perseverance), teamwork 
(adaptability, coordination, decision making, interpersonal, leadership, and 
communication), and creativity (flexibility, elaboration, originality, and fluency). The 
same measures were used for the post test. 

Teamwork- interpersonal (p=.0) and leadership (p=0.2) were considered to be 
statistically significant, creativity- elaboration approached statistical significance 
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(p= .05). Self-efficacy and retention were positively correlated (.27). Creativity 
fluency (-.29) and flexibility (-.34) showed to be negatively correlated to transfer. 

Discussion 

This study proposed to examine the effects of cognitive readiness via a simulation 
intervention. The simulation being the Multi Mission Team Trainer in the area of surface 
warfare was the intervention. The cognitive readiness questionnaire was given before and 
after the Multi Mission Team Trainer. The participants were composed of fifty four 
students and three experts. Results provided by the study provided some evidence 
supporting the hypothesis that students increase their scores from the pre test to the post 
test via simulation intervention. However, the correlations amongst each subscale with 
retention and transfer demonstrated that there were few relationships that were 
statistically significant. Moreover, according to the literature it was expected that 
simulations would increase cognitive readiness. It was expected that the increase of 
simulations will increase content understanding, problem solving skills, and self- 
regulation (O’Neil, in preparation). 

Findings on Problem Solving Measure 

Problem solving measures consisted of two domains, domain independent and 
domain specific. The domain independent measures which consisted of control strategies, 



self-efficacy, effort and perseverance, worry, and elaboration, showed that there was no 
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statistical significant relationship with retention and transfer questions. With the 
exception of the correlation of self-efficacy and retention (which was statistically 
significant. .27), the domain specific questions that were based from Mayer’s retention 
and transfer questions showed that there were no correlation between the measures and 
the rest of the subscales. 

One hypothesis that might explain why there was little relationship between the 
domain independent measures and the retention and transfer questions is because students 
were less than optimally motivated. The day in which the questions were given was after 
students had a Multi-Mission Team Trainer simulation scenario (one and a half hour), 
had taken the post questionnaire (40 minutes), and had to take their final exam for the 
class (2 hours). The final exam was high stakes whereas the Multi Mission Team Trainer 
performance was low stakes. As the participants had chosen the Navy as a career the 
exam was a must and thus high stakes. In contrast, participants knew that the cognitive 
readiness questionnaire and retention and transfer questions were for this research study 
and thus were low stakes. This research data was not provided to the instructors and thus 
could not affect their grade. 

These tasks might have also caused the students to experience cognitive overload. 
Cognitive overload theory is based on a cognitive architecture that consists of a limited 
working memory, with partly independent processing units for visual and audio 



information, which interacts with an unlimited long-term memory (Kirshner, 2001). 
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There are three different types of cognitive load theory, intrinsic, extraneous, 
and germane (Kirshner, 2001). For the purpose of this research, intrinsic and extraneous 
are the relevant and applicable explanations. Intrinsic cognitive load is determined by the 
interaction of the materials under study and the expertise of the learner. One person may 
be able to handle several pieces of information and call on stored schemata to handle one 
or more items (Van Merrienboer & Ayres, 2005). The extraneous load depends on the 
way tasks are presented to the learner. One way to overload the working memory is to 
present a diagram and written word simultaneously. Shifting the written work to spoken 
word relieves the visual memory of some information and the auditory memory assumes 
the burden (van Merrienboer & Ayres, 2005). Thus the retention and transfer data 
probably underestimates participant’s true performance in the Multi-Mission Team 
Trainer. 

Moreover, the measures for problem solving might have been written in a more 
comprehensive and explicit manner. For example, for future research, the retention and 
transfer questions can be incorporated as part to the education program in order for 
students to have gained that prior knowledge tested by retention questions. For future 
research the researcher might want to choose different measures for the domain 
independent problem solving strategies, such as adaptive expertise, resiliency, and 



adaptability. 
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Findings on the Teamwork Measure 

The teamwork measure which was adopted from O’Neil (2005) showed that 
interpersonal and leadership increased significantly after the Multi Mission Team Trainer 
experience. Instructional strategies for teamwork skills training include the use of 
multiple strategies. An instructional strategy to support teamwork might include concepts 
or a kind of strategy where the learner is able to recognize unfamiliar examples of 
procedures or actions belonging to a particular group or category of knowledge 
(Marshall, O’Neil, Chen, Kuehl, and Abedi, 2005). 

The post-test teamwork-interpersonal scale in this study shows having the highest 
means from all subscales. These results were consistent with a study conducted at the 
U.S. Marine Corps (Marshall, O’Neil, Chen, Kuehl, and Abedi, 2005). These findings are 
also consistent with O’Neil who found that interpersonal skills scale had the highest 
mean score of all the scales (Marshall, O’Neil, Chen, Kuehl, and Abedi, 2005). However, 
O’Neil et. al. also concluded that leadership had lowest mean score, which is not the case 
for this study. 

Teamwork is a common desired of many and school environments that use 
collaborative problem solving techniques. The results of this study indicate the 
measurement of teamwork processes can be accomplished in a reasonably reliable and 



much timely manner. 
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Findings on the Creativity Measure 

The measurement for creativity was adopted from Abedi (2006). The elaboration 
subscale approached statistical significance (p=.05). Elaboration was defined as the 
ability to fill in the details. The rest of the subscales were not statistically significant. 
Unexpectedly, were the correlations which demonstrated that the more fluency and 
flexibility the less transfer knowledge performance (which was statistically significant) 
was gained by students. Fluency was defined as the ability to produce large numbers of 
ideas, and flexibility, the ability to produce a variety of ideas or use a variety of 
approaches (Abedi & Schumacher (2002). This result was unexpected. 

The study conducted by Abedi- Shumacher (2002) in Spain provided an 
opportunity to compare the traditional internal consistency and concurrent validity of the 
creativity test. The results of analyses based on the traditional approaches indicated that 
the concurrent validity coefficients were not high. The results also revealed relatively low 
correlations between the academic achievement measures and the creativity measures. 

The mean for elaboration (which consisted of 11 items) was 2.40, the standard deviation 
was 0.28, and the alpha reliability was .61 (Abedi & Schumacher, 2002). 

Abedi (2002) found that creativity-elaboration indicated that creativity and 
academic abilities were related in this sample, but they were not the same or even highly 
correlated constructs. Moreover, it can be concluded from the results of this study to 



modify the creativity test items in order to be more domain specific to the navy setting in 
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new and upcoming studies. Also, a more thorough rubric needs to be developed in 
order to score the creativity scales. The process is two-fold, present a stimulus and have 
participants generate as many possible responses to the stimulus. Scoring involves 
counting popular, unusual/ original scores. 

Findings on Problem Solving Questions (Retention and Transfer ) 

The two questions adapted from Mayer on retention and transfer, were not 
significantly significant with each other. The correlations indicated that student’s 
performance on the different cognitive readiness measures did not correlate well with 
retention and transfer. However, the retention and transfer measures were reliable with 
some validity information. The current Multi Mission Team Trainer has no process or 
outcome performance measures. The measures created for this study could be use for 
assessment in surface warfare and could be created for other areas such as air defense and 
anti submarine warfare. 



Implications 

There are two types of implications for this study, one is research based and the 
other is practice based. One researched based implication is that more constructs of 
cognitive readiness should be tested in order to have a better understanding of the effects 



of cognitive readiness in a surface warfare simulation. Other competencies like 




101 

adaptability, and resiliency, should also be measured in order to create a better 
assessment of the effects of cognitive readiness. 

One possible alternative is to have other measures that are tested to be valid and 
reliable. The study only concentrated on three measures because they were feasible, 
reliable, and valid. The three measures included problem solving adopted from PISA 
(Marsh, 2006), Teamwork adopted from O’Neil (2003), and Creativity adopted from 
Abedi (2002). All the components from the cognitive readiness model should be 
measured. The components include adaptive expertise, creativity, resiliency, situational 
awareness, problem solving, adaptability, decision making, and teamwork. 

In addition, as an initial small-scale investigation, this study provides promise that 
larger scale experimentation may provide more stable and statistically significant 
estimates of the relationship between teamwork measures, such as adaptability, 
coordination, decision making, leadership, and interpersonal process, and outcome 
measure such retention and transfer knowledge via the Multi Team Trainer simulation. 

This study has shown that the use of simulations can be effectively incorporated 
in assessing the relationships among the increase of cognitive readiness. However, more 
retention and transfer scaffolding needs to be in place and aligned with the curriculum, 
feedback, and assessments. For example, there should be more direct instruction on the 
case scenarios and simulations in order for students to build more prior knowledge. In 



order to improve performance at the retention and transfer level, the goals and objectives 
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of the case scenarios must be in alignment with the goals and objectives of the written 
and simulated test. 
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Appendix B 



STUDENT INTRODUCTION 



Thank you for agreeing to participate in our survey for the effects of cognitive readiness 
in a surface warfare simulation. It will take approximately 30-40 minutes to complete the 
questionnaires. Your participation in this study is completely voluntary. Please try to 
answer as many statements as you can. It is very important for us to learn your opinions. 
Your survey responses will be strictly confidential and data from this research will be 
reported only in the aggregate. If you have questions at any time about the survey or the 
procedures, you may contact Ms. Donna Ayala at dayala@usc.edu , Mr. Sutter Fox at 
ffox@usc.edu . Or if you have questions regarding your rights as a research subject, call 
Mr. Sid Kushner of the IRB Board at 213-537-5027. 



Thank you very much for your time and support. 



Appendix C 

Problem Solving Questionnaire 



Service Number:_ 

Directions. A number of statements that people have used to describe themselves are 
given below. Read each statement and indicate how you generally think or feel by 
marking one of the responses. There are no right or wrong answers. Do not spend too 
much time on any one statement. Remember, mark the answer that seems to describe how 
you generally think or feel. 



Almost Never Some-times Often 



1. When I study, I try to relate 1 2 3 

new material to things I have 

learned in other subjects. 

2. Thinking about my grade in a 1 2 3 

course interferes with my work 

on tests. 

3. I’m certain I can understand the 1 2 3 

most difficult material 

presented in texts. 

4. When I study, I start by 12 3 

figuring out exactly what I 

need to learn. 

5. I freeze up on important exams. 1 2 3 

6. When studying, I work as hard 1 2 3 

as possible. 

7. When I study, I figure out how 1 2 3 

the information might be useful 

in the real world. 

8. During exams I find myself 1 2 3 

thinking about whether I’ll ever 

get through school. 

9. I’m confident I can understand 1 2 3 

the most complex material 

presented by the teacher. 



Almost 

Always 

4 

4 

4 

4 

4 

4 

4 

4 

4 
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Almost Never Some-times Often 

10. When I study, I force myself to 1 2 3 

check to see if I remember 

what I have learned. 

11. The harder I work at taking a 1 2 3 

test, the more confused I get. 

12. When studying, I keep working 1 2 3 

even if the material is difficult. 

13. When I study, I try to 1 2 3 

understand the material better 

by relating it 

to things I already know. 

14. Thoughts of doing poorly 1 2 3 

interfere with my concentration 

on tests. 

15. When I study, I try to figure 1 2 3 

out which concepts I still 

haven’t really understood. 

16. I’m confident I can do an 1 2 3 

excellent job on assignments 

and tests. 

17. When studying, I try to do my 1 2 3 

best to acquire the knowledge 

and skills taught. 

18. I seem to defeat myself while 1 2 3 

working on important tests. 

19. When I study, I figure out how 1 2 3 

the material fits in with what I 

have already learned. 

20. I’m certain I can master the 1 2 3 

skills being taught. 



Almost 

Always 

4 



4 

4 

4 



4 

4 

4 

4 

4 

4 

4 
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Almost Never Some-times Often Almost 

Always 

21. During tests I find myself 1 2 3 4 

thinking about the 

consequences of failing. 

22. When I study, I make sure that 1 2 3 4 

I remember the most important 

things. 

23. When studying, I put forth my 1 2 3 4 

best effort. 



24. During examinations I get so 1 2 3 4 

nervous that I forget facts 

I really know. 

25. When I study, and I don’t 1 2 3 4 

understand something I look 

for additional information to 
clarify this. 



SCORING KEY 


Scale/Item 


Item No. 


Elaboration 




When I study, I try to relate new material to things I have learned 
in other subjects. 


1 


When I study, I figure out how the information might be useful 
in the real world. 


7 


When I study, I try to understand the material better by relating it 
to things I already know. 


13 


When I study, I figure out how the material fits in with what I 
have already learned. 


4 



Self Efficacy 
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Scale/Item Item No. 

I’m certain I can understand the most difficult material presented 3 

in texts. 

I’m confident I can understand the most complex material 9 

presented by the teacher. 

I’m confident I can do an excellent job on assignments and tests. 16 

I’m certain I can master the skills being taught. 20 

Effort and Perseverance 

When studying, I work as hard as possible. 6 

When studying, I keep working even if the material is difficult. 12 

When studying, I try to do my best to acquire the knowledge and 17 

skills taught. 

When studying, I put forth my best effort. 23 

Control Strategies 

When I study, I start by figuring out exactly what I need to learn. 14 

When I study, I force myself to check to see if I remember what I 10 

have learned. 

When I study, I try to figure out which concepts I still haven’t 15 

really understood. 

When I study, I make sure that I remember the most important 22 

things. 

When I study, and I don’t understand something I look for 25 

additional information to clarify this. 

Test Anxiety (Worry) 

Thinking about my grade in a course interferes with my work on 2 

tests. 

I freeze up on important exams. 5 

During exams I find myself thinking about whether I’ll ever get 8 

through school. 
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Scale/Item Item No. 

The harder I work at taking a test, the more confused I get. 1 1 

Thoughts of doing poorly interfere with my concentration on 14 

tests. 

I seem to defeat myself while working on important tests. 18 

During tests I find myself thinking about the consequences of 21 

failing. 

During examinations I get so nervous that I forget facts I really 24 

know. 
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Teamwork Questionnaire 
Appendix D 



Service Number: . 

Directions: This set of questions is to help us understand the way you think and feel 
about working with others. Read each statement below and indicate how you generally 
think or feel. There are no right or wrong answers. Do not spend too much time on each 
question. 





Almost 

Never 


Some- 

times 


Often 


Almost 

Always 


1. When I work as part of a 
team, I exercise leadership. 


1 


2 


3 


4 


2. When I work as part of a 
team, I ensure the instructions 
are understood by all team 
members prior to starting the 
task. 


1 


2 


3 


4 


3. When I work as part of a 
team, I understand and 
contribute to the organizational 
goals. 


1 


2 


3 


4 


4. When I work as part of a 
team, I teach other team 
members. 


1 


2 


3 


4 


5. When I work as part of a 
team, I interact cooperatively 
with other team members. 


1 


2 


3 


4 


6. When I work as part of a 
team, I allocate the tasks 
according to each team 
member’s abilities. 


1 


2 


3 


4 


7. When I work as part of a 


1 


2 


3 


4 





team, I know the process of 
making a decision. 










8. When I work as part of a 
team, I know the process of 
making a decision. 


1 


2 


3 


4 


9. When I work as part of a 
team, I conduct myself with 
courtesy. 


1 


2 


3 


4 


10. When I work a part of a 
team, I ask for the instructions 
to be clarified when it appears 
not all the team members 
understand the task. 


1 


2 


3 


4 


11. When I work as part of a 
team, I help ensure the proper 
balancing of the workload. 


1 


2 


3 


4 


12. When I work as part of a 
team, I know how to weigh the 
relative importance among 
different issues. 


1 


2 


3 


4 


13. When I work as part of a 
team, I lead when appropriate, 
mobilizing the group for high 
performance. 


1 


2 


3 


4 


14. When I work as part of a 
team, I respect thoughts and 
opinions of others in the team. 


1 


2 


3 


4 


15. When I work as part of a 
team, I can identify potential 
problems readily. 


1 


2 


3 


4 


16. When I work as part of a 
team, I communicate in a 
manner to ensure mutual 
understanding. 


1 


2 


3 


4 


17. When I work as part of a 
team, I do my part of the 
organization in a timely manner. 


1 


2 


3 


4 


18. When I work as part of a 
team, I prepare sufficiently to 


1 


2 


3 


4 





make a decision. 










19. When I work as part of a 
team, I lead the team 
effectively. 


1 


2 


3 


4 


20. When I work as part of a 
team, I treat others with 
courtesy. 


1 


2 


3 


4 


21. When I work as part of a 
team, I willingly contribute 
solutions to resolve problems. 


1 


2 


3 


4 


22. When I work as part of a 
team, I seek and respond to 
feedback. 


1 


2 


3 


4 


23. When I work as part of a 
team, I track other team 
member’s progress. 


1 


2 


3 


4 


24. When I work as part of 
team, I solicit input for decision 
making from my team 
members. 


1 


2 


3 


4 


25. When I work as part of a 
team, I demonstrate leadership 
and ensure team results. 


1 


2 


3 


4 


26. When I work a part of a 
team, I adapt readily to varying 
conditions and demands. 


1 


2 


3 


4 


27. When I work as part of a 
team, I listen attentively. 


1 


2 


3 


4 


28. When I work as part of a 
team, I am able to change 
decisions based upon new 
information. 


1 


2 


3 


4 


29. When I work as part of a 
team, I try to bring out the best 
in others. 


1 


2 


3 


4 


30. When I work as part of a 
team, I recognize conflict. 


1 


2 


3 


4 


31. When I work as part of a 


1 


2 


3 


4 
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team, I clearly and accurately 
exchange information. 










32. When I work as part of a 
team, I emphasize the meeting 
of deadlines. 


1 


2 


3 


4 


33. When I work as part of a 
team, I accept individual 
differences among members. 


1 


2 


3 


4 


34. When I work as part of a 
team, I identify needs or 
requirements and develop 
quality/timely solutions. 


1 


2 


3 


4 


35. When I work as part of a 
team, I pay attention to what 
other are saying. 


1 


2 


3 


4 


36. When I work as part of a 
team, I treat all my team 
members as equals. 


1 


2 


3 


4 



Teamwork Scoring Key 



Scales 


Items 


Coordination (n=5) 


6,11,17,23,32 


Decision Making (n=6) 


3,7,12,18,24,28 


Leadership (n=7) 


1,4,8,13,19,25,29 


Interpersonal Skills (n=6) 


5,9,14,20,33,36 


Adaptability (n=5) 


15, 21, 26, 30, 34 


Communication (n=7) 


2, 10, 16, 22, 27,31,35 
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Creativity Questionnaire 
Appendix E 



Service Number: 

Directions: This set of questions is to help us understand your level of creativity in 
different situations. Read each statement below and bubble in one answer per question. 
Do not spend too much time on each question. 

ELABORATION SUBSCALE 

1) When something unfair happens to you, can you figure out all of the reasons 
behind that event? 

O I don't. 

O Sometimes, I do. 

O Usually, I do. 

2) When something strange occurs, what do you usually do? 

O I do not look for the possible reasons. 

O I look for some possible reasons. 

O I look for all of the possible reasons. 

3) When you get interested in something, how much attention do you pay 
to the details? 

O I do not pay much attention to the details. 

O I pay attention to some of the details. 

O I pay attention to all of the details. 

4) When you listen to a song, how much attention do you pay to what 
the song says? 

O I never pay attention. 
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O Sometimes, I pay attention. 

O I pay a lot of attention. 

5) When you look at a piece of art work, do you think about what the artist 
is trying to say? 

O I do not. 

O I do at times. 

O I always do this. 

6) After you have seen a movie that has affected you, what do you 
usually do? 

O I move on to my next activity. 

O I think about some aspects of the movie. 

O I think about many aspects of the movie. 

7) When you write a letter, what do you usually include? 

O I include some general information about myself. 

O I include some general information and a few additional description. 

O I include some general information and many additional description. 

ELABORATION SUBSCALE ( continued ) 

7) When you read a book, do you create pictures of the characters 
in your mind? 

O I do not create pictures. 

O I create some pictures. 

O I create many pictures. 

9) How much complexity do you enjoy in what you do? 

O I enjoy doing things that are straightforward. 

O I enjoy a little complexity in what I do. 

O I enjoy a lot of complexity in what I do. 
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10) How involved do you get in the details of what you are doing? 

O I do not get involved with the details. 

O I get involved with a few of the details. 

O I get involved with many of the details. 

FLEXIBILITY SUBSCALE 

11) Would you ever be interested in a job which would require you to 
make up stories in front of audiences? 

O I would not be interested. 

O I might be interested. 

O I would be interested. 

12) When you need to communicate with someone who speaks English poorly, 
how easy is it for you to come up with simpler ways to communicate with that 
person? 

O It usually is difficult for me. 

O It sometimes is easy for me. 

O It often is easy for me. 

13) How much do you enjoy making new things out of existing objects? 

O I usually do not enjoy making new things. 

O I sometimes enjoy making new things. 

O I often enjoy making new things. 

14) What do you do with people who are difficult to convince? 

O I have trouble coming up with convincing arguments. 

O I try to come up with few alternative arguments to convince them. 

O I come up with a variety of arguments to convince them. 

15) How do you approach a complex task? 

O I come up with a single approach. 
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O I may be able to come up with few approaches. 

O I will be able to come up with a variety of approaches. 

16) Which kind of job you like the most? 

O I l ik e a job in which almost everything is done based on rules 
or instructions. 

O I l ik e a job in which some of the work is done based on rules 
O or instruction. 

O I l ik e a job which is not based on rules or instructions. 

17) What kind of job interests you? 

O A job which requires you to follow instruction. 

O A job which requires some talent. 

O A job which requires a lot of talent. 

FLEXIBILITY SUBSCALE (continued) 

18) If you were invited to a meeting to discuss problems in your community, 

how difficult it would be for you to come up with a long list of problems? 

O It would be very difficult. 

O It would somewhat difficult. 

O It would not be difficult at all. 

19) If you were asked to help the city council to come up with many ideas, 

how would you do? 

O It would be difficult for me to generate ideas beyond those the City Council has 
considered already. 

O I would be able to generate a few ideas beyond those which had been considered 
already. 

O I would be able to generate a variety of ideas for the City Council. 
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20) Would you be able to help a first grade teacher generate many alternative 
ways for teaching numbers in such a way that children would become interested in 
learning numbers. 

O No, I would not be able to do that. 

O I might be able to generate a few alternative ways. 

O Yes, I would be able to generate many alternative ways. 

21) How easy would it be for you to help a school with very limited resources, 

to come up with interesting ideas for new sports and games? 

O It would be very difficult. 

O I would be able to come up with few ideas. 

O I would be able to come up with many ideas. 

22) How easily do words come to you when you express yourself? 

O Words do not come easily. 

O Sometimes, words come easily. 

O Often, words come easily. 

23) How well do you express yourself? 

O I do not express myself well. 

O Occasionally, I express myself well. 

O Often, I express myself well. 
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FLUENCY SUBSCALE 



24) How well do you express your ideas? 

O I have difficulty expressing my ideas well. 

O I am able to express some of my well. 

O I am able to express all of my ideas well. 

25) If a group of people were to ask you to give an unprepared speech, 

how well do you do? 

O I would not do well. 

O I am not sure how well I would do. 

O I would do well. 

26) How easy is it for you to come up with words to describe something? 
O It is difficult for me. 

O It is not very easy for me. 

O It is easy for me. 

27) How comfortable would you be in a job which requires you to come 

up with many ideas? 

O I would not be comfortable. 

O I might be comfortable. 

O I would be very comfortable. 

28) How easy would it be for you to come up with many alternative 

words to express the same idea? 

O It would be quite difficult for me. 

O It would not that easy for me. 

O It would be quite easy for me. 

29) If you had to participate in a contest in which you were asked to come 
up with as many words as possible which began with the letter "J", how 
would you do? 

O I would do poorly. 




129 

O I would do okay. 

O I would do very well. 

30) If you were asked to participate in a contest of naming things that belong in a 
given category (class) like food or plants, how well would you do? 

O I would do poorly. 

O I would do well. 

O I would do quite well. 

FLUENCY SUBSCALE (continued) 

31) How many sentences could you come up if they all would have to begin 

with the same word? 

O I could come up with only a few sentences. 

O I could come up with several sentences. 

O I could come up with many sentences. 

32) Would you be able to come up with a number of possible uses for 
things other than their common use? 

O It would be very difficult for me. 

O I might be able to come up with a few possible uses. 

O I would be able to come up with a number of possible uses. 

33) How easy would it be for you to tell many new stories? 

O It would be difficult for me. 

O It would not be that easy for me. 

O It would be easy for me. 

34) Which of the following would you prefer to do? 

O Reading a poem. 

O Learning a poem. 

O Discussing a poem. 
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35) If you were asked to write a newspaper article about your community, which 
of the following choices would you prefer? 

O I would copy other articles. 

O I would add to other articles 

O I would write my own article. 

36) Which of the following do you enjoy the most? 

O I enjoy reading famous books. 

O I enjoy reading famous books and writing some of my own. 

O I enjoy writing my own books. 

37) How easy would it be for you to write many synonyms for the word "fast"? 
O It would be very difficult for me. 

O It would be somewhat difficult for me. 

O It would be easy for me. 

FLUENCY SUBSCALE (continued) 

38) Let us suppose that we lost all of the possible ways to heat food, how easy 
would it be for you to come up with a list of other ways to heat food? 

O It would be very difficult for me. 

O It would be somewhat difficult for me. 

O It would be easy for me. 

39) If you had to give a speech, how would you present it? 

O I would read my prepared notes verbatim. 

O I would read mostly from my notes. 

O I would glance occasionally at my notes. 
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40) When you are faced with an extremely difficult problem, what do you usually 
do? 

O I usually ask for the answer. 

O I usually look up for the answer. 

O I usually make up my own answer. 

ORIGINALITY SUBSCALE 

41) If You were making something, and you discovered suddenly that you were 
missing an important part, what would you do? 

O I would stop. 

O I would try to find the missing parts, and if I could not find them, I would 
stop. 

O I would try to find the missing parts, and if I could not find them, I would 
make some substitute. 

42) When you are in a public place, do you ever try to guess what people who 
are far away from you are discussing? 

O I am never interested in guessing what other people discuss. 

O Sometimes I like to guess what other people discuss. 

O I always like to guess what other people are discussing. 

43) Do you enjoy solving difficult problems? 

O I do not enjoy solving difficult problems. 

O I rarely enjoy solving difficult problems. 

O I often enjoy solving difficult problems. 

44) When you are faced with a new kind of problem, what do you usually 
do? 

O I have someone else do it for me. 

O I try to do it with someone else's help. 

O I try to get more information so that I can solve it on my own. 
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45) What would you do if you were solving a difficult problem? 

O I would ask a teacher or someone else to help me. 

O I would read a book on that subject. 

O I would try a number of different ways to come up with my own answer. 

46) When you have an unusual problem, how do you usually solve it? 

O I often seek help. 

O Before I seek help, I spend a short period of time trying to solve the problem 
on my own. 

O I spend a lot of time trying to solve a problem on my own. 

ORIGINALITY SUBSCALE (continued) 

47) Do people think that you ask tough questions? 

O No, they don't. 

O Sometimes, they do. 

O Often, they do. 

48) Do people think that you come up with unique ideas? 

O No, they don't. 

O Sometimes, they do. 

O Often, they do. 

49) Do you find that you usually like to try new things? 

O I do not usually try new things. 

O I sometimes try new things. 

O I often try new things. 

50) When you encounter a complex problem what do you do? 

O I try to avoid it. 

O I spend a short period of time trying to solve it until I get frustrated. 




o 



I spend a lot of time trying to solve it. 



51) Do you enjoy new experiences? 

O I do not enjoy new experiences. 

O I sometimes enjoy new experiences. 

O I usually enjoy new experiences. 

52) How much do you depend on others when you find yourself in 
impossible situation? 

O I often depend on others. 

O I sometimes depend on others. 

O I usually prefer to depend on myself. 

53) How confident are you when you do something on you own? 
O I do not have much confidence. 

O I have some confidence. 

O I have a lot of confidence. 

ORIGINALITY SUBSCALE (continued) 

54) Do you enjoy scientific experimentation? 

O I do not enjoy experimentation. 

O I enjoy experimentation to some extent. 

O I enjoy experimentation a lot. 

55) How good would you be at designing new toys for children? 

O I would not be very good at this. 

O I might be able to come up with a few ideas. 

O I would be able to come up with many ideas. 
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56) If you were a substitute teacher for a Kindergarten class without a teaching 
plan, how well would you do? 

O I would fail. 

O I would have a hard time. 

O I would do well. 
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Appendix F 

Table 34: Student’s Retention Measure (N=52) 




101 1 

104 1 1 1 11 1 

105 1 1 

106 1 1 

107 1 1 1 

108 1 1 1 1 1 1 1 

109 1 

110 

111 1 111 1 

112 1 

113 

114 

115 111111 11 

116 1 1 1 1 111 

117 1 1 1 

118 1 1 1 1 1 1 

119 1 1 1 1 1 1 



1 

6 

2 

2 

3 

7 

1 

5 

1 



8 

7 

3 

6 

6 
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120 

121 1 1 1 

122 1 1 

124 

125 1 1 

126 1 1 

127 1 1 

129 1 1 

130 

131 1 1 

132 1 1 1 

134 

135 

136 

138 1 1 

139 

141 1 

142 

144 1 1 

145 1 1 

147 1 1 1 

148 1 

149 1 1 

159 1 1 1 

160 
161 

162 1 1 

163 1 1 

164 1 1 

165 1 

166 

167 1 1 

168 1 1 

169 1 

170 1 

TOTALS 35 28 15 



1 

1 

1 1 
1 

1 1 

1 

1 

1 



1 

6 17 



111 



1 



1 

1 



1 1 



8 4 4 8 



1 

7 

3 

2 

2 

2 

5 

5 

3 



2 

2 

1 

4 

2 

4 

1 

2 

6 



2 

2 

2 

1 

2 

2 

1 

2 

126 
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Appendix G 

Table: 35- Retention Individual Raters Score 



Stud. No. Rater 1 Rater 2 

101 1 

104 5 

105 1 

106 1 

107 2 

108 7 

109 1 

110 0 

111 4 

112 1 

113 0 

114 0 

115 8 

116 7 

117 3 

118 6 

119 6 

120 1 

121 7 

122 3 

124 0 

125 2 

126 2 

127 2 

129 2 

130 0 

131 5 

132 3 

134 0 

135 0 

136 0 

138 2 

139 2 

141 1 

142 0 

144 4 

145 2 

147 3 

148 1 



Consensus 

1 

6 

1 

1 

3 

6 

1 

0 

5 
1 
0 
0 
8 
7 
2 

6 
6 
1 
7 

3 
0 
2 
2 
2 
2 
0 

4 

3 
0 
0 
0 
2 
2 
1 
0 

4 
2 
4 
1 



1 

6 

1 

1 

3 

7 
1 
0 

5 
1 
0 
0 

8 
7 
3 

6 
6 
1 
7 
3 
0 
2 
2 
2 
2 
0 
5 

3 
0 
0 
0 
2 
2 
2 
0 

4 
2 
4 
1 
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149 

159 

160 
161 
162 

163 

164 

165 

166 

167 

168 

169 

170 



2 

6 

0 

0 

2 

2 

2 

1 

0 

2 

2 

1 

2 



2 

6 

0 

0 

2 

2 

2 

1 

0 

2 

2 

1 

2 



2 

6 

0 

0 

2 

2 

2 

1 

0 

2 

2 

1 

2 
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Appendix H 



Table 36: Student’s Transfer Measure (N=52) 




101 

104 1 

105 

106 

107 1 

108 1 

109 

110 1 
111 

112 1 

113 1 

114 1 

115 

116 



1 



11 1 11 



1 1 



1 



1 



1 



2 



1 

6 

3 

1 

2 

1 

1 

1 



TOTAL 
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117 

118 

119 

120 
121 
122 

124 

125 

126 
127 

129 

130 

131 

132 

134 

135 

136 

138 

139 

141 

142 

144 

145 

147 

148 

149 

159 

160 
161 
162 

163 

164 

165 

166 

167 

168 

169 

170 



1 1 
1 



1 




1 

1 



1 



1 

1 

1 1 
1 



1 



1 



1 

20 7 



1 

1 




1 



1 



1 

1 

1 



1 



1 

6 



1 

4 2 2 2 



2 

2 



2 

2 

3 

2 



1 

1 

1 

1 

1 

1 

2 

3 

2 



1 

1 

1 



6 



1 

2 

50 
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Appendix I 

Table 37: Transfer Individual Raters Score 



Stud. 

No. Rater 1 Rater 2 



101 1 2 

104 0 0 

105 0 0 

106 0 0 

107 1 1 

108 6 5 

109 0 0 

110 3 3 

111 0 0 

112 1 1 

113 2 2 

114 1 1 

115 1 1 

116 1 1 

117 0 0 

118 2 1 

119 0 0 

120 1 1 

121 0 0 

122 0 0 

124 1 1 

125 0 0 

126 1 1 

127 2 3 

129 1 1 

130 0 0 

131 0 0 

132 0 0 

134 0 0 

135 0 0 

136 1 1 

138 1 1 

139 1 1 

141 1 1 



Consensus 

2 

0 

0 

0 

1 

6 

0 

3 

0 

1 

2 

1 

1 

1 

0 

2 

0 

1 

0 

0 

1 

0 

1 

3 

1 

0 

0 

0 

0 

0 

1 

1 

1 

1 




142 



142 1 1 

144 1 1 

145 1 1 

147 0 0 

148 2 2 

149 3 2 

159 1 1 

160 1 1 

161 0 0 

162 1 1 

163 0 0 

164 0 0 

165 1 1 

166 1 1 

167 0 0 

168 0 0 

169 1 1 

170 2 2 



1 

1 

1 

0 

2 

2 

1 

1 

0 

1 

0 

0 

1 

1 

0 

0 

1 

2 




